Text Analysis

The module allows the calculation of some textual and semantic metrics for the documents in the corpus. A brief description is provided in the following sections.

Analyses

Basic textual metrics are calculated at every run (such as the number of types and tokens for each document), even if no other analysis is selected. In addition, you can choose to carry out these other more advanced analyses:

• Complexity: calculates the language complexity of each document. The function provides several metrics: the number of words of six or more letters (absolute and relative frequencies), the average word length, and other complexity scores calculated using the TF-IDF function and considering the word frequency distribution. The function also calculates the numerical intensity and readability of the text (Gunning-Fog index).
• Sentiment: calculates the average language sentiment of each document. Scores can vary from -1 to 1, where positive values indicate positive sentiment.
• Emotions partial: calculates several dimensions of the language used (such as the degree of positive and negative emotions or the language orientation towards the past or future). Scores are normalized considering the length of the document and can range from 0 to 100. To obtain the raw scores, you can multiply values by WordCountOriginal and divide by 100.
• Emotions full: calculates additional emotions, such as anger or joy, as well as scores for valence, dominance, and arousal. These scores are inspired by the NRC lexicon (for more information, see https://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm). Valence, dominance, and arousal can be positive or negative (without a predefined range). The other emotion scores are normalized considering the length of the document and can range from 0 to 100. To obtain the raw scores, you can multiply by WordCountOriginal and divide by 100.
• Crovitz: calculates the relative frequencies of the Crovitz’s relational words that appear in each document.

Parameters

• Csv separator: specifies the separator used in the CSV file. Insert a single character without quoting.
• Percentage of text to analyze: used when the analysis has to consider only a portion of each text document, for example, just the title and lead of online news. A value of 1 means that the entire text will be analyzed; lower values indicate a lower percentage of text to analyze (e.g., 0.5 = the initial 50% of each document).
• Language: this is the language of uploaded texts (please be consistent and try to analyze one language at a time).
• Choose one or more analyses: select the analyses you want to carry out. This field is optional: if you leave it empty, the output will contain only the basic textual metrics, plus the brand and dimension columns if you specified them.

Optional parameters

• Dimensions: this field can be used to specify custom dimensions/dictionaries, to count how many of their words appear in each document. You can use a dictionary to represent each dimension. The following syntax has to be used "dimension_name1":["word1","word2",..], "dimension_name2":["word6","word8",..],... The dimension name will not be considered as a word for the analysis. Each dimension produces three columns in the output (“_count”, “_perc” and “_diff”, see Output); when you fill in this field, the app also shows a reminder of how they are calculated.
  • Hyphens and whitespaces determine what a word matches: a word with a hyphen, such as “zero-emission”, counts “zero-emission”, “zero emission” and “zeroemission”; a word with a whitespace, such as “zero emission”, counts “zero emission” and “zero-emission”; a joined word, such as “zeroemission”, counts only “zeroemission”. Hyphens can only be used inside words (not at the beginning or end of a word, not doubled, and not next to a whitespace or an asterisk). Words are never counted inside longer words, but they are counted inside hyphenated words (e.g., “emission” is also counted in “zero-emission”, but not in “zeroemission”).
  • Asterisks can be used at the end of a word, indicating that the word could be completed with any possible set of characters: "asp*" matches “aspirin” and “aspire”. Asterisks also work at the end of expressions of multiple words: "ant colon*" matches “ant colony”, “ant colonies” and “ant-colony”, while "ant-colon*" also matches the joined forms, such as “antcolony”. Only one asterisk can be used, at the end of the word or expression and not after a whitespace (“*” alone, “ant *” or “a*b*” return an error). Asterisks also decide how words are counted in the “_diff” columns (see Output).
  • Repeated words: the same word can be used in different dimensions, but words in the same dimension cannot overlap. The software returns an error if, in the same dimension, a word is repeated, written differently (hyphens and whitespaces are ignored in this check, so “self-driving”, “self driving” and “selfdriving” are the same word), or already covered by a word with an asterisk (e.g., “ant colon*” and “ant colony”, or “self-driv*” and “self-driving”). An asterisk at the end of a single word, such as “machine*”, does not cover expressions with a whitespace, such as “machine learning”.
• Case sensitive (Dimensions): if selected, the words of the dimensions are matched case sensitively (e.g., “Apple” will not count “apple”).
• Brand list: use this field if you want to count how many times a brand is mentioned in each document and then average the overall results by brand. A brand can be any word, such as the name of a product or a person. You must list the brands without quotes, separating them with a comma. Brands cannot contain hyphens: hyphenated words are joined during the text cleaning (e.g., “coca-cola” becomes “cocacola”), so write the brand without the hyphen, or use a Word Cluster such as "cocacola":["coca-cola"] to match “coca-cola”, “coca cola” and “cocacola”.
• Word Clusters: sometimes, it is useful to merge multiple words representing a common concept/brand. Each brand/concept could be represented by a set of keywords. If this is the case, you can use the cluster brands field to specify the words to merge. All words in a cluster will be replaced with the cluster label.
  For example, we may want to have a single word in lieu of the word “Pope” and the word “Francis”. The following syntax has to be used "cluster1":["word1","word2",..], "cluster2":["word6","word8",..],...
  • Hyphens and whitespaces determine what a cluster word matches: a word with a hyphen, such as “self-driving”, matches “self-driving”, “self driving” and “selfdriving”; a word with a whitespace, such as “self driving”, matches “self driving” and “self-driving”; a joined word, such as “selfdriving”, matches only “selfdriving”. Hyphens can only be used inside words (not at the beginning or end of a word, not doubled, and not next to a whitespace or an asterisk). Cluster words are never matched inside longer words: for example, “car” will not change “cars” or “car-sharing”. Hyphenated words that are not in a cluster are joined during the text cleaning (e.g., “mother-in-law” becomes “motherinlaw”).
  • Asterisks can be used at the end of a word, indicating that the word could be completed with any possible set of characters: "asp*" matches “aspirin” and “aspire”. Asterisks also work at the end of expressions of multiple words: "ant colon*" matches “ant colony”, “ant colonies” and “ant-colony”, while "ant-colon*" also matches the joined forms, such as “antcolony”. Only one asterisk can be used, at the end of the word or expression and not after a whitespace (“*” alone, “ant *” or “a*b*” return an error).
  • Repeated words: the same word cannot appear in multiple clusters, not even written differently: hyphens and whitespaces are ignored in this check, so “self-driving” in one cluster and “selfdriving” or “self driving” in another will return an error (different spellings of the same word can be listed in the same cluster). Likewise, a word with an asterisk cannot cover another word, in the same cluster or in another one: “ant colon*” together with “ant colony” or “ant-colony”, or “self-driv*” together with “self-driving”, will return an error. An asterisk at the end of a single word, such as “machine*”, does not cover expressions with a whitespace, such as “machine learning”.
  • Cluster order: the word order within each cluster will not impact substitutions, but the cluster order will. For example, if you input the text "I like machine learning" and provide clusters in the order "x1":["machine learning"], "x2":["machine", "learning"] the substitution result will be "I like x1". If you provide clusters in the order "x2":["machine", "learning"], "x1":["machine learning"], the software will return an error (substitution result would be "I like x2 x2").
• Case sensitive (Word Clusters): if selected, Word Clusters substitutions will be case sensitive.
• Time unit and Time intervals: use this field if you also want to average results by time intervals. If the “full dataset” option is selected, the analysis will run on all the text documents without considering time intervals.
• Start time and End time are the analysis’s start and end times.

Output

Documents are analyzed following their order in the input file. The function will produce the following files:

• “TextAnalysis.csv”: this is the main file with results. A brief explanation of its columns is provided below. Please note that not all columns listed here might appear in the file, depending on the parameters you set: the columns of an analysis appear only if that analysis was selected (the “Complexity” analysis also includes “Emotions partial”, which is needed for the numerical intensity):
  • “Date” is the document date.
  • “WordCountOriginal” is the number of words (tokens) in the original document before text preprocessing.
  • “WordCountClean” is the number of words (tokens) in the clean document after text preprocessing.
  • “Types” is the number of different unique word stems.
  • “LexicalDiversity” refers to the ratio of different unique word stems (“Types”) to the total number of words (“WordCountClean”).
  • “Hapaxes” is the number of words that appear only once within a document. For example, the sentence “Sun, look at the sun!” will have 3 hapaxes, 4 types, and 5 tokens.
  • If brands have been specified in the parameters, a series of columns labeled with their names will appear. The values in these columns indicate how many times each brand is mentioned in each document.
  • “Sentiment” is a measure of the language sentiment, calculated for the entire document. Scores will differ from those of sentiment calculated specifically for brands (see the main analysis). The calculation of sentiment also differs depending on the language of the documents (for example, we use the VADER lexicon for English). Scores range from -1 (negative) to +1 (positive).
  • “LoughranSentimentNEG”, “LoughranSentimentPOS”, “LoughranUncertainty”, “LoughranModalStrong” and “LoughranModalWeak” columns report the scores obtained by using the Loughran-McDonald lexicon, which proved useful for the calculation of uncertainty and positive and negative sentiment in financial contexts (see here and here). The lexicon also includes dictionaries for weak and strong modals. Scores represent the total frequency of words of each dictionary appearing in the text, divided by the document length and multiplied by 100. This function is only available for some languages.
  • “AvgWlen” is the average length of words in the documents, measured by the average number of characters.
  • “SixLtrs” and “SixLtrs_perc” are the number of words in the text that have six or more letters and their ratio to the total number of words in the document.
  • “GunningFog” is the Gunning-Fog index, used to measure the readability of a text document.
  • “NumDigits” is the count of numbers (expressed with digits) appearing in the text, divided by the document length (number of words) and multiplied by 100.
  • “NumIntensity” is the sum of the values in the columns labeled “NumDigits”, “Number”, and “NumOperations”. This dimension considers the amount of quantitative information provided in the document — counting numerical terms (including integers, numbers in lexical format, and terms referring to numerical operations) and dividing this number by the total word count. See this paper for more info and a use case.
  • “ComplexityStDev” is a measure of language complexity calculated as the standard deviation of the frequency counts of the words appearing in a document. You might consider normalizing this value to take the document length into account.
  • “ComplexityTFIDFsum” is a measure of the informativeness of the text, calculated following a TF-IDF logic. The idea is that a text is more informative if it has more words that deviate from the common language. Longer texts will get higher informativeness scores. The formula used is the same as “ComplexityTFIDFavg”, without the 1/n term.
  • “ComplexityTFIDFavg” is a measure of the average complexity/distinctiveness of the text, calculated following a TF-IDF logic. The idea is that a document is distinctive if it contains words that do not commonly appear in the other documents and if these words are not lost in uninformative text blobs. See this paper and look at the “Measuring linguistic distinctiveness” section for more information and the formula.
  • Regarding the columns “valence”, “arousal” and “dominance”, please refer to the documentation of the NRC-VAD lexicon. The software calculates scores for these three dimensions, which can be negative or positive (without a predefined range). The contribution of each word appearing in a document is summed up, considering a rescaling of the values of the original lexicon (from [0,1] to [-1,1]).
  • For columns from “AngerNRCCount” to “sadnessIntensity” please refer to the documentation of the NRC lexicon. The software calculates scores for the emotions of anger, anticipation, disgust, fear, joy, sadness, surprise, and trust expressed in the text. It will also show the aggregate counts of words representing positive and negative emotions. Please consider that, for each emotion, you will get two scores: the first one (“Count”) represents the percentage of words in the text related to each emotion, with values from 0 to 100; the second one (“Intensity”) considers the intensity of words (e.g., attributing a higher intensity score to the word “magnificent” than to the word “pretty”).
  • For columns from “Posemo” to “NumOperations” please refer to the documentation of the LIWC software and its related publications. The SBS BI app tries to map dimensions similar to those mapped by LIWC but using different dictionaries. We kept the labeling scheme to support researchers already familiar with it. Scores represent the total frequency of words of each dimension appearing in the text, divided by the document length and multiplied by 100. Please note that the names of these columns might change depending on the language chosen for the analysis. Only to understand the labeling, please refer to the LIWC software dictionaries and manual.
  • Columns named “Promotion” and “Prevention” refer to the construct of Regulatory Focus, with dictionaries inspired by the work of Gamache et al. (2015). Scores represent the total frequency of words of each dimension appearing in the text, divided by the document length and multiplied by 100. Please note that the availability of these columns might change depending on the language chosen for the analysis.
  • Columns from “about” to “with” represent the frequency count of the Crovitz’s relational words that appear in each text document. Scores can vary from 0 to 100. Please note that column names will change depending on the language chosen for the analysis (the original list is in English). This function is only available for some languages.
  • The file will end with three columns for each custom dimension you might have specified in the parameters. Columns are labeled with the name you assigned to the dimension followed by a suffix. For example, for a dimension named “quality” you will get:
    • “quality_count”: the number of times the words of the dimension appear in the text. Every occurrence counts, including repeated ones; an expression of multiple words, such as “ant colony”, counts as one occurrence.
    • “quality_perc”: the value of “quality_count” divided by “WordCountOriginal” (the number of words in the original document, before text preprocessing) and multiplied by 100.
    • “quality_diff”: the number of different words of the dimension that appear at least once in the text, regardless of how many times each of them appears. For example, if the word “great” appears five times and no other word of the dimension appears, “quality_count” will be 5 and “quality_diff” will be 1. If three different words of the dimension appear, “quality_diff” will be 3.
    Please note that, in the “_diff” columns, what counts is each word as you wrote it in the dimension, together with all the forms it matches. A word with an asterisk, such as “ant colon*”, counts once even if the text contains both “ant colony” and “ant colonies”. If instead you list “ant colony” and “ant colonies” as two separate words, they count as two different words. If singular and plural forms have to count as one word, write them with an asterisk. Similarly, the word “self-driving” counts once whether the text contains “self-driving”, “self driving” or “selfdriving”. For example, the text “An ant colony is not two ant colonies” has 8 words (“WordCountOriginal”) and gives:

    Dimension	_count	_perc	_diff
    "ants1":["ant colon*"]	2	25	1
    "ants2":["ant colony","ant colonies"]	2	25	2

• “TextAnalysisBrandsTime.csv”: this is the same as the previous file, with results averaged by brand and/or time intervals, if these were specified (for example, “quality_count” becomes the average number of occurrences per document).