Loading...
Unsupervised joint PoS tagging and stemming for agglutinative languages
Bolucu, Necva Can, Burcu
Bolucu, Necva
Can, Burcu
Authors
Editors
Other contributors
Affiliation
Epub Date
Issue Date
2019-01-25
Submitted date
Alternative
Abstract
The number of possible word forms is theoretically infinite in agglutinative languages. This brings up the out-of-vocabulary (OOV) issue for part-of-speech (PoS) tagging in agglutinative languages. Since inflectional morphology does not change the PoS tag of a word, we propose to learn stems along with PoS tags simultaneously. Therefore, we aim to overcome the sparsity problem by reducing word forms into their stems. We adopt a Bayesian model that is fully unsupervised. We build a Hidden Markov Model for PoS tagging where the stems are emitted through hidden states. Several versions of the model are introduced in order to observe the effects of different dependencies throughout the corpus, such as the dependency between stems and PoS tags or between PoS tags and affixes. Additionally, we use neural word embeddings to estimate the semantic similarity between the word form and stem. We use the semantic similarity as prior information to discover the actual stem of a word since inflection does not change the meaning of a word. We compare our models with other unsupervised stemming and PoS tagging models on Turkish, Hungarian, Finnish, Basque, and English. The results show that a joint model for PoS tagging and stemming improves on an independent PoS tagger and stemmer in agglutinative languages.
Citation
Bölücü, N. and Can, B. (2019) Unsupervised joint PoS tagging and stemming for agglutinative languages. ACM Transactions on Asian and Low-Resource Language Information Processing 18 (3): 25. DOI: 10.1145/3292398
Research Unit
DOI
PubMed ID
PubMed Central ID
Embedded videos
Additional Links
Type
Journal article
Language
en
Description
This is an accepted manuscript of an article published by Association for Computing Machinery (ACM) in ACM Transactions on Asian and Low-Resource Language Information Processing on 25/01/2019, available online: https://doi.org/10.1145/3292398
The accepted version of the publication may differ from the final published version.
Series/Report no.
ISSN
EISSN
2375-4702
ISBN
ISMN
Gov't Doc #
Sponsors
This research is supported by the Scientific and Technological Research Council of Turkey (TUBITAK) with the project number EEEAG-115E464.