Corpus Cleaner

This function can be used to pre-process and clean the documents, with the option to remove stop-words and perform language stemming or lemmatization. It can also be used to substitute words in the text and to calculate corpus statistics. In particular, the software will calculate: the number of Tokens and Types, the Type-Token Ratio, the number of Hapaxes, and the Hapax-Type Ratio.

Parameters

• Csv separator: specifies the separator used in the CSV file. Insert a single character without quoting.
• Percentage of text to retain: useful to retain only the initial part of each text document, for example, just the title and lead of online news. A value of 1 means that the full text will be analyzed; lower values indicate a lower percentage of text to analyze (e.g., 0.5 = the initial 50% of each document).
• Stemming or lemmatization: choose one of the two operations. You can also skip stemming or lemmatization while choosing the language.
• Language for stopwords: this is used to load language-based stop-word dictionaries. Choose “SKIP” to avoid removing stop-words.
• Language for stemming/lemmatization: indicates the language used for stemming or lemmatization. Choose “SKIP” to avoid both operations.
• Custom stopwords: can be used to specify custom stopwords, i.e., words that will be ignored during the analysis. List custom stopwords separated by a comma, without quotes. Including multiple words (e.g., "formula 1") is possible.
• Word Clusters: sometimes, it is useful to merge multiple words representing a common concept. Each concept could be represented by a set of keywords. If this is the case, you can use the cluster word substitution field to specify the words to merge. All words in a cluster will be replaced with the cluster label.
  For example, we may want to have a single word in lieu of the word “Pope” and the word “Francis”. The following syntax has to be used "cluster1":["word1","word2",..], "cluster2":["word6","word8",..],... The same word cannot appear in multiple clusters. Hyphens cannot be used in words in the cluster. Please replace them with a whitespace (e.g., if you want to replace the word “zero-emission”, please put “zero emission” in the cluster). Additionally, asterisks can be used at the end of words, indicating that a specific word could be completed with any possible set of characters. For example, if the word "asp*" is used, this will match both the words "aspirin" and "aspire". This does not work with multiple words.
  Please note that the word order within each cluster will not impact substitutions, but the cluster order will. For example, if you input the text "I like machine learning" and provide clusters in the order "x1":["machine learning"], "x2":["machine", "learning"] the substitution result will be "I like x1". If you provide clusters in the order "x2":["machine", "learning"], "x1":["machine learning"], the software will return an error (substitution result would be "I like x2 x2").
• Case sensitive: if selected, Word Clusters substitutions will be case sensitive.