Explaining Contextualization in Language Models using Visual Analytics
R. Sevastjanova, A. Kalouli, C. Beck, H. Schäfer, und M. El-Assady. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Seite 464--476. Online, Association for Computational Linguistics, (August 2021)
DOI: 10.18653/v1/2021.acl-long.39
Zusammenfassung
Despite the success of contextualized language models on various NLP tasks, it is still unclear what these models really learn. In this paper, we contribute to the current efforts of explaining such models by exploring the continuum between function and content words with respect to contextualization in BERT, based on linguistically-informed insights. In particular, we utilize scoring and visual analytics techniques: we use an existing similarity-based score to measure contextualization and integrate it into a novel visual analytics technique, presenting the model's layers simultaneously and highlighting intra-layer properties and inter-layer differences. We show that contextualization is neither driven by polysemy nor by pure context variation. We also provide insights on why BERT fails to model words in the middle of the functionality continuum.
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
%0 Conference Paper
%1 sevastjanova-etal-2021-explaining
%A Sevastjanova, Rita
%A Kalouli, Aikaterini-Lida
%A Beck, Christin
%A Schäfer, Hanna
%A El-Assady, Mennatallah
%B Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
%C Online
%D 2021
%I Association for Computational Linguistics
%K 2021 d02 sfbtrr161
%P 464--476
%R 10.18653/v1/2021.acl-long.39
%T Explaining Contextualization in Language Models using Visual Analytics
%U https://aclanthology.org/2021.acl-long.39
%X Despite the success of contextualized language models on various NLP tasks, it is still unclear what these models really learn. In this paper, we contribute to the current efforts of explaining such models by exploring the continuum between function and content words with respect to contextualization in BERT, based on linguistically-informed insights. In particular, we utilize scoring and visual analytics techniques: we use an existing similarity-based score to measure contextualization and integrate it into a novel visual analytics technique, presenting the model's layers simultaneously and highlighting intra-layer properties and inter-layer differences. We show that contextualization is neither driven by polysemy nor by pure context variation. We also provide insights on why BERT fails to model words in the middle of the functionality continuum.
@inproceedings{sevastjanova-etal-2021-explaining,
abstract = {Despite the success of contextualized language models on various NLP tasks, it is still unclear what these models really learn. In this paper, we contribute to the current efforts of explaining such models by exploring the continuum between function and content words with respect to contextualization in BERT, based on linguistically-informed insights. In particular, we utilize scoring and visual analytics techniques: we use an existing similarity-based score to measure contextualization and integrate it into a novel visual analytics technique, presenting the model{'}s layers simultaneously and highlighting intra-layer properties and inter-layer differences. We show that contextualization is neither driven by polysemy nor by pure context variation. We also provide insights on why BERT fails to model words in the middle of the functionality continuum.},
added-at = {2022-04-25T08:59:56.000+0200},
address = {Online},
author = {Sevastjanova, Rita and Kalouli, Aikaterini-Lida and Beck, Christin and Sch{\"a}fer, Hanna and El-Assady, Mennatallah},
biburl = {https://puma.ub.uni-stuttgart.de/bibtex/298bd63d5d004a85417214a04473a26dc/christinawarren},
booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
doi = {10.18653/v1/2021.acl-long.39},
interhash = {f0b053765f0ab9fd13f6722b645f4e83},
intrahash = {98bd63d5d004a85417214a04473a26dc},
keywords = {2021 d02 sfbtrr161},
month = aug,
pages = {464--476},
publisher = {Association for Computational Linguistics},
timestamp = {2022-04-25T06:59:56.000+0200},
title = {Explaining Contextualization in Language Models using Visual Analytics},
url = {https://aclanthology.org/2021.acl-long.39},
year = 2021
}