View source for Lemmatization

{{Short description|Natural language processing canonicalisation}}
'''Lemmatization''' (or less commonly '''lemmatisation''') in [[Social:Linguistics|linguistics]] is the process of grouping together the [[Social:Inflection|inflected form]]s of a word so they can be analysed as a single item, identified by the word's [[Social:Lemma (morphology)|lemma]], or dictionary form.<ref>''Collins English Dictionary'', entry for "lemmatize"</ref>

In [[Computational linguistics|computational linguistics]], lemmatization is the algorithmic process of determining the [[Social:Lemma (morphology)|lemma]] of a word based on its intended meaning. Unlike [[Stemming|stemming]], lemmatization depends on correctly identifying the intended part of speech and meaning of a word in a sentence, as well as within the larger [[Social:Context (language use)|context]] surrounding that sentence, such as neighbouring sentences or even an entire document. As a result, developing efficient lemmatization algorithms is an open area of research.<ref name="Semantic Annotation Research">{{cite web |url=https://scholar.archive.org/work/wes6yl773jb7jb3w5jvdydiuei|title=WebBANC: Building Semantically-Rich Annotated Corpora from Web User Annotations of Minority Languages}}</ref><ref name="Muller, University of Munich">{{cite conference  |url=http://www.cis.lmu.de/~muellets/pdf/emnlp_2015.pdf|title=Joint Lemmatization and Morphological Tagging with LEMMING|first1=Thomas|last1=Müller|first2=Ryan|last2=Cotterell|first3=Alexander|last3=Fraser|first4=Hinrich|last4=Schütze|date=2015|doi=10.18653/v1/D15-1272|doi-access=free|publisher=Association for Computational Linguistics|conference=2015 Conference on Empirical Methods in Natural Language Processing|pages=2268–2274|location=Lisbon}}</ref><ref>{{Cite web|url=http://homepages.inf.ed.ac.uk/s1044253/papers/Context_Sensitive_Neural_Lemmatization_with_Lematus.pdf|title=Context Sensitive Neural Lemmatization with Lematus|last1=Bergmanis|first1=Toms|last2=Goldwater|first2=Sharon|}}</ref>

==Description==
In many languages, words appear in several ''inflected'' forms. For example, in English, the verb 'to walk' may appear as 'walk', 'walked', 'walks' or 'walking'. The base form, 'walk', that one might look up in a dictionary, is called the ''lemma'' for the word. The association of the base form with a part of speech is often called a ''[[Social:Lexeme|lexeme]]'' of the word.

Lemmatization is closely related to [[Stemming|stemming]]. The difference is that a stemmer operates on a single word ''without'' knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech. However, stemmers are typically easier to implement and run faster. The reduced "accuracy" may not matter for some applications. In fact, when used within information retrieval systems, stemming improves query [[Precision and recall#Recall|recall accuracy]], or true positive rate, when compared to lemmatization. Nonetheless, stemming reduces [[Precision and recall#Recall|precision]], or the proportion of positively-labeled instances that are actually positive, for such systems.<ref name="Stanford Information Retrieval Book">{{cite web |url=http://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html|title=Introduction to Information Retrieval|publisher=Cambridge University Press|first1=Christopher D.|last1=Manning|first2=Prabhakar|last2=Raghavan|first3=Hinrich|last3=Schütze}}</ref>

For instance:
#The word "better" has "good" as its lemma. This link is missed by stemming, as it requires a dictionary look-up.
#The word "walk" is the base form for the word "walking", and hence this is matched in both stemming and lemmatization.
#The word "meeting" can be either the base form of a noun or a form of a verb ("to meet") depending on the context; e.g., "in our last meeting" or "We are meeting again tomorrow". Unlike stemming, lemmatization attempts to select the correct lemma depending on the context.

Document indexing software like [[Software:Apache Lucene|Lucene]]<ref name="Lucene Snowball" >{{cite web
|url=http://lucene.apache.org/core/
|title=Lucene Snowball
|publisher=Apache project
}}</ref> can store the base stemmed format of the word without the knowledge of meaning, but only considering word formation grammar rules. The stemmed word itself might not be a valid word: 'lazy', as seen in the example below, is stemmed by many stemmers to 'lazi'. This is because the purpose of stemming is not to produce the appropriate lemma&nbsp;– that is a more challenging task that requires knowledge of context. The main purpose of stemming is to map different forms of a word to a single form.<ref name="Porter Stemmer" >{{cite web
|url=http://tartarus.org/~martin/PorterStemmer/
|title=Porter Stemmer
|author=Martin Porter
}}</ref> As a rule-based algorithm, dependent only upon the spelling of a word, it sacrifices accuracy to ensure that, for example, when 'laziness' is stemmed to 'lazi', it has the same stem as 'lazy'.

==Algorithms==
A trivial way to do lemmatization is by simple dictionary lookup. This works well for straightforward inflected forms, but a [[Rule-based system|rule-based system]] will be needed for other cases, such as in languages with long compound words. Such rules can be either hand-crafted or learned automatically from an annotated [[Text corpus|corpus]].

==Use in biomedicine==
Morphological analysis of published biomedical literature can yield useful results. Morphological processing of biomedical text can be more effective by a specialized lemmatization program for biomedicine, and may improve the accuracy of practical [[Information extraction|information extraction]] tasks.<ref>{{Cite journal | last1 = Liu | first1 = H. | last2 = Christiansen | first2 = T. | last3 = Baumgartner | first3 = W. A. | last4 = Verspoor | first4 = K. | title = BioLemmatizer: A lemmatization tool for morphological processing of biomedical text | doi = 10.1186/2041-1480-3-3 | journal = [[Medicine:Journal of Biomedical Semantics|Journal of Biomedical Semantics]] | volume = 3 | pages = 3 | year = 2012 | pmid =  22464129| pmc =3359276 | doi-access = free }}</ref>

==See also==
* [[Canonicalization]]

==References==
{{reflist}}

==External links==

{{Natural Language Processing}}

[[Category:Computational linguistics]]

de:Lemma (Lexikografie)#Lemmatisierung

{{Sourceattribution|Lemmatization}}