Year: 2022
Type: dissertation
Abstract: The treatment of mental disorders nowadays entails a wide variety of still non-solved \ntasks such as misdiagnosis or delayed diagnosis. During this doctoral thesis we study and \ndevelop different models that can serve as potential tools for the clinician labor. Among \nour proposals, we outline two main lines of research, Natural Language Processing and \nprobabilistic methods. \nIn Chapter 2, we start our thesis with a regularization mechanism used in language \nmodels and specially effective in Transformer-based architectures, where we call it NoRBERT, \nfrom Noisy Regularized Bidirectional Representations from Transformers [9], [15]. \nAccording to the literature, we found out that regularization in NLP is a low explored \nfield limited to the use of general mechanisms such as dropout [57] or early stopping \n[58]. In this landscape, we propose a novel approach to combine any LM with Variational \nAuto-Encoders [23]. VAEs belong to deep generative models, with the construction of \na regular latent space that permits the reconstruction of the input samples throughout an \nencoder and decoder networks. Our VAE is based in a prior distribution of a mixture \nof Gaussians (GMVAE), what gives the model the chance to capture some multimodal \ninformation. Combining both, Transformers and GMVAEs we build an architecture capable \nof imputing missing words from a text corpora in a diverse topic space as well as \nimprove BLEU score in the reconstruction of the data base. Both results depend on the \ndepth of the regularized layer from the Transformer Encoder. The regularization in essence \nis formed by the GMVAE reconstruction of the Transformer embeddings at some point in \nthe architecture, adding structure noise that helps the model a better generalization. We \nshow improvements in BERT[15], RoBERTa [16] and XLM-R [17] models, verified in \ndifferent datasets and we also provide explicit examples of sentences reconstructed by \nTop NoRBERT. In addition, we validate the abilities of our model in data augmentation, \nimproving classification accuracy and F1 score in various datasets and scenarios thanks \nto augmented samples generated by NoRBERT. We study some variations in the model, \nTop, Deep and contextual NoRBERT, the latter based in the use of contextual words to \nreconstruct the embeddings in the corresponding Transformer layer. \nWe continue with the Transformers line of research in Chapter 3, proposing PsyBERT. \nPsyBERT, as the own name refers, is a BERT-based [15] architecture suitably modified \nto work in Electronic Health Records from psychiatry patients. It is inspired by BEHRT [19], also devoted to EHRs in general health. We distinguish our model from the training \nmethodology and the embedding layer. In a similar way that with NoRBERT, we find \nthe utility of using a Masked Language Modeling (MLM) policy without no finetuning or \nspecific-task layer at all. On the one hand, we used MLM in NoRBERT to solve the task \nof imputing missing words, finishing the aim of the model in generating new sentences by \ninputs with missing information. On the other hand, we firstly propose the use of PsyBERT \nsuch as tool to fill the missing diagnoses in the EHR as well as correct misdiagnosed \ncases. After this task, we also apply PsyBERT in delusional disorder detection. On the \ncontrary, in this scenario we apply a multi-label classification layer, that aims to compute \nthe probability of the different diagnoses in the last visit of the patient to the hospital. \nFrom these probabilities, we analyse delusional cases and propose a tool to detect potential \ncandidates of this mental disorder. In both tasks, we make use of several fields obtained \nfrom the patient EHR, such as age, sex, diagnoses, treatments of psychiatric history and \npropose a method capable of combining heterogeneous data to help the diagnosis in mental \nhealth. During these works, we point out the problematic in the quality of the data from \nthe EHRs [104], [105] and the great advantage that medical assistance tools like our \nmodel can provide. We do not only solve a classification problem with more than 700 \ndifferent illnesses, but we bring a model to help doctors in the diagnosis of very complex \nscenarios, with comorbidity, long periods of patient exploration by traditional methodology \nor low prevalence cases. We present a powerful method treating a problematic with great \nnecessity. \nFollowing the health line of research and psychiatry application, we analyse in Chapter \n4 a probabilistic method to search for behavioral pattern in patients also with mental \ndisorders. In this case it is not the method the contribution of the work but the application \nand results in collaboration with the clinician interpretation. The model is called SPFM \n(Sparse Poisson Factorization Model) [22] and consist on a non-parametric probabilistic \nmodel based on the Indian Buffet Process (IBP) [20], [21]. It is a exploratory method \ncapable of decomposing the input data in sparse matrixes. For that, it imposes the Poisson \ndistribution to the product of two matrixes, Z and B, both obtained respectively by the IBP \nand a Gamma distribution. Hence Z corresponds to a binary matrix representing active \nlatent features in a patient data and B weights the contribution of the data characteristics to \nthe latent features. The data we use in the three works described during the chapter refers \nto different questions from e-health questionnaries. Then, the data characteristics refer to \nthe answer or punctuation on each question and the latent features from different behavioral \npatterns in a patient regarding the selection of features active in their questionnaires. For \nexample, patient X can present feature 1 and 2 and patient Y may presence feature 1 \nand 3, giving as a result two different profiles of behavioral. With these procedure we \nstudy three scenarios. In the first problematic, we relate the profiles with the diagnoses, \nfinding common patterns among the patients and connections between diseases. We also \nanalyse the grade of critical state and contrast the clinician judgment via the Clinical \nGlobal Impression (CGI). In the second scenario, we pursue a similar study and find \nout connections between disturbed sleeping patterns and clinical markers of wish to die. We focus this analysis in patients with suicidal thoughts due to the problematic that \nthose individuals suppose as a major public health issue [175]. In this case we vary \nthe questionnarie and the data sample, obtaining different profiles also with important \ninformation to interpret by the psychiatrist. The main contribution of this work is the \nproportion of a mechanism capable of helping with detection and prevention of suicide. \nFinally, the third work comprehend a behavioral pattern study in mental health patient \nbefore and during covid-19 lockdown. We did not want to lose the chance to contribute \nduring coronavirus disease outbreak and presented a study about the changes in psychiatric \npatients during the alarm state. We analyse again the profiles with the previous e-health \nquestionnaire and discover that the self-reported suicide risk decreased during the lockdown. \nThese results contrast with others studies [237] and suppose signs for an increase in suicidal \nideation once the crisis ceases. \nFinally, Chapter 5 propose a regularization mechanism based in a theoretical idea from \n[245] to obtain a variance reduction in the real risk. We interpret the robust regularized \nrisk that those authors propose in a two-step mechanism formed by the minimization of the \nweighted risk and the maximization of a robust objective and suggest an idea to apply this \nmethodology in a way to select the samples from the mini-batch in a deep learning set up. \nWe study different variations of repeating the worst performed samples from the previous \nmini-bath during the training procedure and show proves of improvements in the accuracy \nand faster convergence rates of a image classification problem with different architectures \nand datasets.
Author: Cobo Aguilera, Aurora
Institution: Universidad Carlos III de Madrid
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7042463613
