Work

Understanding Data Scientists’ use of Explainability and Interpretability tools

Year: 2021
Type: dissertation
Abstract: Machine learning techniques are being used increasingly in high-risk domains such as healthcare, law, and finance. Misclassifications or mispredictions in areas like these can have serious consequences. Therefore, it is crucial to have high performing models that can make as minimal errors as possible. However, in some cases, it is not enough to know what the predictions and classifications are, it is also important to understand why a given model is making certain decisions. Explainability and interpretability tools help data scientists in understanding how an output is obtained by a machine learning model from a given input. Model explainability and interpretability tools like SHAP (Shapley Additive Explanations) and GAMs (Generalised additive models) are becoming widely used. However, user studies that evaluate the extent to which these tools help data scientists interpret and explain their models are still uncommon. Our study attempts to find out how data scientists use these tools – their general behaviour, factors affecting their trust in the tools, and their expectations from the tools. We conduct a qualitative study consisting of Pilot interviews and Contextual inquiries. By analyzing the results, the study shows that the tools sometimes are not used at their full potential. Moreover, some participants expressed the need to also align with other stakeholders to get the full picture on the explanations and trust them. The most important factors affecting their trust were the lack of clarity in understanding the visualisations that further lead to the participants reasoning intuitively and having suspicions about the explanations,
Author: Baghele, Purvesh Kamlesh (author)
Language: English
Open Access status: green
OpenAlex ID: W7071833178
Cites: 46
Cited by:
Related to: 10