Text File Upload

The app can analyze any textual data. A corpus should be uploaded as a single CSV file with the following characteristics:

• Has no header.
• Is fully encoded in utf-8.
• Text has to be in the first column, and dates (yyyy-mm-dd) in the second column. No other date formats are accepted.
• Text must have no quoting, and the CSV separator must not be in the text field.
• An optional third column can contain either source weights (numeric values; higher scores indicate more important sources) or a label in the format LABEL_PERIOD or LABEL_PERIOD_CATEGORY, where PERIOD is an integer and LABEL/CATEGORY are strings. The format must be consistent across all rows.
• Each user has a maximum allowed file size. Uploading bigger files is not allowed: they will be automatically deleted after upload.

You can analyze only one file at a time. The uploader will retain only the last uploaded CSV file, deleting the others. Uploading a new file also deletes the results (and the plot) of the previous validation.

Validate Text File

This function is used to validate the uploaded CSV file and verify that there are no errors. Please:

• specify the CSV separator used in the CSV file (use a single character with no quoting) and the minimum length of documents you want to consider (they need to have at least three characters). Maximum document length is 2,000,000 characters;
• optionally choose to validate the third column using the Validate third column dropdown: select Source weights to verify that the third column contains numeric values, or LABEL_PERIOD or LABEL_PERIOD_CATEGORY to verify that it follows one of those two label formats consistently across all rows. Select Do not validate to skip this check;
• optionally choose to automatically remove lines with errors. Please be careful, as this will change the uploaded file;
• optionally choose to remove or keep all the text documents (file rows) that match a specific search query. This will significantly slow down the validation process. The following parameters can be used:
  • Use search operators: this option can be used to remove or keep only the documents that match the search query specified through the following fields. If the “No” option is selected, the system will only identify and list the documents matching the query without keeping or removing them. A preliminary run with the “No” option selected is recommended to see which (and how many) documents are matched by the query.
  • Doc includes ALL these words: matched documents will include ALL the words listed here. List words with no quoting or spacing, only separated by a comma. Use lowercase and only letters (no punctuation).
  • Doc includes AT LEAST ONE of these words: matched documents will include AT LEAST ONE of the words listed here. List words with no quoting or spacing, only separated by a comma. Use lowercase and only letters (no punctuation).
  • Doc DOES NOT INCLUDE ANY of these words: matched documents will NOT include any of the words listed here. List words with no quoting or spacing, only separated by a comma. Use lowercase and only letters (no punctuation).
• optionally choose to keep only the documents within a date range, using the Start date and End date fields (extremes included). All the lines with a date outside the range are removed from the file, even if Remove lines with errors is not selected; they are not checked further and are not counted among errors or search matches. You can fill only one of the two fields for an open-ended range (e.g., all documents from a given date onwards), or leave both empty to keep all dates. Lines with an invalid date are errors and are removed only if Remove lines with errors is selected. If no line falls within the range, the date filter is not applied and the file is not emptied.

Output

The validated (and corrected) file will be automatically saved on the system (replacing the original CSV). There is no need to reupload a correctly validated file. The system will also offer the possibility to download the input file that has been corrected.

The validation message also reports the number of documents per year, counting only the lines that remain in the file after the validation (and after the optional removal of lines with errors, search query and date range filters). After the validation, the Documents per Year Plot button opens the same information as an interactive bar chart in a new browser tab: hover over a bar to see its exact value, and use the camera icon in the top-right corner of the chart to download it as a PNG image. Years with no documents are shown as zero; lines without a valid date are not shown in the chart (their number is reported in the chart subtitle).

Data Management

The "Delete All My Data" button allows users to permanently erase all user data and saved reports from the system. Please use this feature with caution, as the deletion is irreversible and cannot be undone.