Psychiatric Disorders With Speech

Mental health is important at every stage of life, from childhood and adolescence through adulthood, even more in the digital world created due to the ongoing pandemic and lockdown. Although we have huge medical research on these disorders, there is no lab test to diagnose any of the disorders. Moreover, early detection of degenerative disease can prevent irreversible changes if detected at a later stage during EEG, MRI, etc. In today's world of voice automation, virtual classes, and meetings, speech data is the easiest sample that can be collected from anyone, using existing devices like phones, laptops.

In this project, we wish to identify psychiatric disorders through patient's speech

Review paper : https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7042657/

List of articles of interest : https://tinyurl.com/y6ojfq56

Task 1 : Import the data from Mobile Device Voice Recordings at King's College London (https://zenodo.org/record/2867216#.X5-_tNtS-gQ). Create a PyTorch dataloader for the dataset and layout procedure for adding more datasets in future.

Task 2 : Create scripts to visiualize the data using matplotlib, or any other python package. Implement any Machine Learning algorithm on the loaded dataset to define training and evaluation flow.

Task 3 : Reproduce the results from https://www.ijraset.com/fileserve.php?FID=31054

Task 4 : Implement any Machine Learning algorithm with or without(end-to-end) feature engineering . To start off, use https://github.com/espnet/espnet.

Task 5 : Create pytorch data loader for all the speech datasets listed below. (about 20)

Task 6 : Convert the speech dataset to words for NLP analysis.

We wish to explore state of the art RNN architectures (GRU, LSTM, seq-to-seq, etc.), transfer learning and meta-learning (learning to learn) approaches for this task. These algorithms would be deployed in real time, low power mobile devices for identifying mental disorders from voice-calls, meetings, etc.

Speech-nlp-datasets

Links to publicly available datasets for modeling health outcomes using speech and language :

Speech-based Corpora

• [Corpus] Speech Database of Typical Children and Children with SLI
  Contains 103 children that are native Czech speakers with specific language impairment. (Grill et al., 2016)

• [Corpus] mPower Study, Parkinson's Disease Data
  Contains audio recorings of 800+ subjects with Parkinson's disease (+ controls) performing a structured mobile phone based test composed of voice, walking, tapping, and memory. Data collection study was performed by (Bot et al., (2016))

• [Corpus] Distress Analysis Interview Corpus
  Contains 189, 20-min long interviews of individuals speaking to a virtual agent. The corpus contains binary and multi-class labels for the severity of depression. The dataset contains audio recordings and features, text transcript, and facial features. The corpus was developed by Gratch et al., (2014) and featured in the Audio Visual Emotion Challenge (AVEC) 2016, 2017

• [Corpus] Oxford LSVT Voice Rehabilitation Data Set
  Contains 14 subjects with Parkinson's Disease used to evaluate whether voice rehabilitation improves phonation. (Tsanas et al., (2014))

• Spanish Parkinson Corpus (contact authors for corpus?)
  Contains 50 subjects with varying severity of Parkinson's, speaking Spanish. Corpus was first presented by (Arroyave et al., (2014)) and subsequently featured in the Interspeech 2015 Computational Paralinguistics Challenge.

• [Corpus] Parkinson Speech Dataset with Multiple Types of Sound Recordings Data Set
  Contains audio recordings from 40 subjects (including 20 control) generating sounds accordings to a transcript (sustained vowel, numbers, short sentences, words) from Turkey. (Sakar et al., (2013))

• [Corpus] Mobile Device Voice Recordings at King's College London (MDVR-KCL) from both early and advanced Parkinson's disease patients and healthy controls
  (Jaeger et al. (2019), doi:10.5281/zenodo.2867216))

• [Corpus] Dem@Care
  Dataset that contains audio, video, physiologic signals of Greek dementia patients in the lab or their home. (Factsheet)

• [Corpus] TORGO Databse
  Contains speech and articulatory data on 7 subjects with Cerebral Palsy or Amyotrophic Lateral Sclerosis. (Rudzicz et al., (2010))

• Child Pathological Speech Database (CPSD) (contact authors for corpus?)
  Contains speech recordings from 99 children on the autism spectrum or language impairmet (specific or not).
  Original paper describing the corpus by Ringeval et al., (2010) and was also made available for the Interspeech 2013 Computational Paralinguistic Challenge.

• [Corpus] Oxford Parkinson's Telemonitoring Dataset
  Monitoring of 42 people with early-stage Parkinson's disease recruited to a six-month trial of a telemonitoring device for remote symptom progression monitoring. (Tsanas et al., (2009))

• [Corpus] Oxford Parkinson Dataset
  Contains recordings from 31 individuals. (Little et al., (2007))

• [Corpus] Saarbruecken Voice Database
  A collection of speech recordings from more than 2,000 people following a transcript of pronouncing vowels and a sentence. Each recording has an associated EEG signal. A subset of the speakers have a pathology (e.g. Laryngitis, Parkinson's disease). Citation: Barry, W. J., & Pützer, M. (2007). Saarbrucken voice database. Institute of Phonetics, Universität des Saarlandes, http://www. stimmdatenbank. coli. uni-saarland. de.
  Example work: Martínez et al., 2012

• [Corpus] ALS Voice Data Set
  Contains voice recordings of 54 speakers, with 39 healthy speakers (23 males, 16 females) and 15 ALS patients with signs of bulbar dysfunction (6 males, 9 females). (Vashkevich et al., (2019))

TalkBank Project

• [Corpus] CHILDES Database
  Contains speech of children with different conditions (e.g. Autism, Down's syndrome, hearing impairment) and across different languages (e.g. English, Dutch, Greek, Mandarin).
  MacWhinney, B. (2014). The CHILDES project: Tools for analyzing talk, Volume II: The database. Psychology Press.

• [Corpus] DementiaBank (from TalkBank)
  Contains recordings of individuals with dementia across different languages. Includes around 400 subjects, most notable in size and containing control subjects is:

  • English Pitt: Longitudinal neuropsychological assessments of 319 subjects (dementia + control) performing Cookie Theft, Word Fluency, Story Recall, and Sentence Construction task. (Becker et al., 1994)

• [Corpus] Clinical TalkBank
  In addition to DementiaBank, TalkBank contains:

  • RHDBank individuals with Right-Hemisphere Disorder
  • TBIBank individuals with Traumatic Brain Injury
  • AphasiaBank a communication disorder affecting ability to speak, write, and understand language due to some trauma to language parts of the brain.
  • FluencyBank contains individuals with language disfluencies due to being a second language learner, or due to stuttering.

Text-based Corpora

• [Corpus] MIMIC III (Medical Information Mart for Intensive Care)
  Contains medical details and outcomes of 40,000+ patients (e.g. demographics, vital signs, laboratory tests, medications) as well as 2M+ free-text written medical notes from medical personnel (e.g. physicians, nurses, etc.). (Johnson et al., (2016)).

• i2b2/UTHealth NLP Task (contact authors for corpus?)
  Contains emergency medical records for 296 patients at Partners HealthCare and medical discharge and correspondance notes between medical personnel. Kumar et al., (2014) describes how the data was processed, and Stubbs et al. (2014) describes the 2014 task of identifying risk factors for heart disease over time.

• Nun Study (contact authors for corpus?)
  Diaries of 93 nuns to used to evaluate cognitive impairment (Alzheimer's disease) in later life. Also contains neuropsychology tests and autopsy information. Study was authored by (Snowdon et al.,(1996))

Aspects to explore

1. Acoustic features independent of context. Pitch, pauses, pronounciation etc.
2. Language features : The patterns present in speech-text converted corpus.