PC-KIMMO> load rules c:\work\englex\english.rul

PC-KIMMO> load lexicon c:\work\englex\english.lex

The lex file has INCLUDE statements for all the separate files of nouns, verbs, affixes, etc. Then PC-KIMMO is ready to use in batch or interactive mode, as clearly documented in Antworth's book.

==================================================================================

==================================================================================

This abstract was pasted into a blank Microsoft Word document, where punctuation was removed, case converted to lower, and hard returns inserted between words, resulting in a file of words, one per line, in the same order as in the text. This file was saved in txt format to a file named ppar1.txt and run as a batch through the "recognize" function of PC-KIMMO with the output logged to a file named pck1.txt, as follows.

PC-KIMMO> log c:\work\output1.txt

PC-KIMMO> file recognize c:\work\pck1.txt

The section of pck1.txt representing the article's title is shown here:

================================================================================

===============================================================================

A couple of observations can be made already. First, PC-KIMMO does not do stemming on roots it does not recognize (i.e., that are not the lexicon files) even if the affix is very common; e.g. dehydrotrametenolic and preadipocyte. This means we would either have to enrich the lexicon files with a lot of scientific terminology or change the affix recognition algorithm to be independent of the root. In my autoencoder, stemming is independent of the root, but must be controlled for unwanted anomalies by excepting certain cases (e.g. basement). The idea of exceptions might not fit easily into the logic of PC-KIMMO.

Second, recognition of a compound surface form produces two lexical form outputs, one with the affixes themselves and one with the implied part-of-speech "gloss", e.g.,

Since we are interested mainly in term conflation, the most direct approach would be to use this output to find the root of each word, then normalize all words to their roots in situ in the text before proceeding with further text analysis (such as term statistics or proximity mapping). The gloss form enables this most easily because the root is identifiable as the only morpheme that begins with a lower case letter.

However, in some cases, multiple gloss forms yielded multiple roots, such as differ and different for the surface form differentiation. Having to make manual choices might be prohibitively labor intensive. In this experiment I took the first root, whatever it was.

*(Clearly, there are more direct ways of doing this. This would be a good C programming exercise for me someday!)

===============================================================================

===============================================================================

In this Medline record there were 224 words, of which 162 (72%) were recognized by PC-KIMMO/ENGLEX, of which 59 (26% of 224) resulted in stemming to a shorter root, of which only 11 (5% of 224) were conflated with another occurrence of the same root. The first two percentages would likely remain stable no matter how many Medline records were pooled, but the last one would be expected to rise with increasing size of the text corpus.

In the following table I have juxtaposed the original text (in blue straight Roman font) with the normalized text (in black italic Arial font) and tried to highlight (in red italic Arial) the problems with the results. Problems are only flagged once but may reoccur (except a big problem with "non-" is flagged twice). See the text following the table for a discussion.

Row 2: One might want to stem chemical affixes such as "dehydro-" and "-ic acid", especially the latter, since the acid and anionic ("-ate") forms are virtually interchangeable in biochemistry. Prefixes are usually dangerous stemming candidates (see below "non-" and "anti-") but "pre-" seems pretty safe; preadipocytes are definitely conflatable with adipocytes in this record. It might not be a good idea in this domain to conflate "sensitiz-" to "sense" which refers to the plus strand of DNA (the minus strand is "antisense"). The same goes for "depend-" to "pend". "non-" should definitely NOT be stemmed under any circumstances for obvious reasons. In this case doing so conflates two entirely different diseases, NIDDM and IDDM.

Row 4: We might question the conflation of "department" to "part" or even to "depart", and also "research" to "search". On the other hand, the seemingly obvious conflation of "medical" and "medicinal" to the same root is not accomplished by PC-KIMMO/ENGLEX. "-ology" would be a desirable suffix to stem in thgis domain.

Row 5,6,8: More questionable prefix-based conflations: "recent" to "cent", "discover" to "cove", "promote" to "mote", and "examine" to "mine".

Row 7: Another total prefix crash, "product" to "duct" – these have distinct meanings in this domain (e.g. tear ducts). Organism scientific names at the family level of taxonomy are often based on a "type genus" name with the same root – e.g. Polyporaceae and Polyporus – and conflating the two levels can be useful.

Row 9: "hyper-" (but not "hypo-") seems like another save prefix to stem; certainly hyperglycemia = glycemia.

Row 14: Pretty uncomfortable conflating "receptor" to "receive". Receptors are super-important in this domain, almost a sacred word.

Row 15: Failure to conflate scientific acronyms. Perhaps can be fixed by enriching ENGLEX abbreviation file, but note my comment in report #1: where is the link in that file, since the spelled out reference is a comment?

Row 17: One last big prefix crash, "important" to "port" (portal means liver in this domain).

To sum up, using PC-KIMMO and ENGLEX out-of-the-box for free-text term conflation in the biomedical/pharmaceutical research domain does not seem very promising. At a minimum the negative and reversive prefix sections of the affix.lex control file need to be commented out. Other possible next steps include enriching the lexicon files with scientific terminology, such as the suffixes from my autoencoder stemmer or terms from the Merck dictionaries. Many of those terms are phrases, however, and it might be very difficult to separate the nouns, verbs, adjectives, etc.