Corpus Cleaner

This function can be used to pre-process and clean the documents, with the option to remove stop-words and perform language stemming or lemmatization. It can also be used to substitute words in the text and to calculate corpus statistics. In particular, the software will calculate: the number of Tokens and Types, the Type-Token Ratio, the number of Hapaxes, and the Hapax-Type Ratio.

Parameters

• Csv separator: specifies the separator used in the CSV file. Insert a single character without quoting.
• Percentage of text to retain: useful to retain only the initial part of each text document, for example, just the title and lead of online news. A value of 1 means that the full text will be analyzed; lower values indicate a lower percentage of text to analyze (e.g., 0.5 = the initial 50% of each document).
• Stemming or lemmatization: choose one of the two operations. Stemming and lemmatization can be skipped by selecting the appropriate option.
• Language for stopwords: this is used to load language-based stop-word dictionaries. Choose “SKIP” to avoid removing stop-words.
• Language for stemming/lemmatization: indicates the language used for stemming or lemmatization. Choose “SKIP” to avoid both operations.
• Custom stopwords: can be used to specify custom stopwords, i.e., words that will be ignored during the analysis. List custom stopwords separated by a comma, without quotes. Including multiple words (e.g., "formula 1") is possible. Custom stopwords cannot contain hyphens: hyphenated words are joined during the text cleaning, so write them joined (e.g., “email” instead of “e-mail”). Asterisks can be used at the end of a word or expression, indicating that it could be completed with any possible set of characters: asp* removes “aspirin” and “aspire”, and ant colon* removes “ant colony” and “ant colonies” (“ant-colony” becomes “antcolony” during the cleaning: use antcolon* to remove it). Only one asterisk can be used, at the end of the word or expression and not after a whitespace (a single “*” returns an error).
• Word Clusters: sometimes, it is useful to merge multiple words representing a common concept. Each concept could be represented by a set of keywords. If this is the case, you can use the cluster word substitution field to specify the words to merge. All words in a cluster will be replaced with the cluster label.
  For example, we may want to have a single word in lieu of the word “Pope” and the word “Francis”. The following syntax has to be used "cluster1":["word1","word2",..], "cluster2":["word6","word8",..],...
  • Hyphens and whitespaces determine what a cluster word matches: a word with a hyphen, such as “self-driving”, matches “self-driving”, “self driving” and “selfdriving”; a word with a whitespace, such as “self driving”, matches “self driving” and “self-driving”; a joined word, such as “selfdriving”, matches only “selfdriving”. Hyphens can only be used inside words (not at the beginning or end of a word, not doubled, and not next to a whitespace or an asterisk). Cluster words are never matched inside longer words: for example, “car” will not change “cars” or “car-sharing”. Hyphenated words that are not in a cluster are joined during the text cleaning (e.g., “mother-in-law” becomes “motherinlaw”).
  • Asterisks can be used at the end of a word, indicating that the word could be completed with any possible set of characters: "asp*" matches “aspirin” and “aspire”. Asterisks also work at the end of expressions of multiple words: "ant colon*" matches “ant colony”, “ant colonies” and “ant-colony”, while "ant-colon*" also matches the joined forms, such as “antcolony”. Only one asterisk can be used, at the end of the word or expression and not after a whitespace (“*” alone, “ant *” or “a*b*” return an error).
  • Repeated words: the same word cannot appear in multiple clusters, not even written differently: hyphens and whitespaces are ignored in this check, so “self-driving” in one cluster and “selfdriving” or “self driving” in another will return an error (different spellings of the same word can be listed in the same cluster). Likewise, a word with an asterisk cannot cover another word, in the same cluster or in another one: “ant colon*” together with “ant colony” or “ant-colony”, or “self-driv*” together with “self-driving”, will return an error. An asterisk at the end of a single word, such as “machine*”, does not cover expressions with a whitespace, such as “machine learning”.
  • Cluster order: the word order within each cluster will not impact substitutions, but the cluster order will. For example, if you input the text "I like machine learning" and provide clusters in the order "x1":["machine learning"], "x2":["machine", "learning"] the substitution result will be "I like x1". If you provide clusters in the order "x2":["machine", "learning"], "x1":["machine learning"], the software will return an error (substitution result would be "I like x2 x2").
• Case sensitive: if selected, Word Clusters substitutions will be case sensitive.