Training Classification Models with Rosette

Note that the number of defined categories will determine the scope of your effort later on. For instance, if you decide to include thousands of categories in your model, it will be cumbersome to sort through all the possible labels when it comes time to manually annotate documents or perform error analysis. We recommend keeping your model as small as possible while still capturing the distinctions you need within your problem domain. If you discover similar categories, consider combining them. Conversely, If your initial categories are too broad, you may need to split them up. You should aim for enough categories in your model to all possibilities within your problem domain. While it may seem convenient, we don't recommend including a catch-all "misc" category within your model.

For some domains, a classification problem may lend itself to a taxonomic model with a hierarchy of categories. It is important to consider such hierarchical relationships when specifying your model. Given the hierarchical relationships, do you care about the upper levels of the hierarchy, or only the more specific categories? For instance, one standard taxonomy developed by the Interactive Advertising Bureau is the IAB Tech Lab Taxonomy. The following is a small subset of the IAB taxonomy:

Depending on the depth of your taxonomy, you need to determine at what level of granularity to cut off your model. At a certain point, the distinctions between categories may be too subtle to reliably determine. For instance, considering the subset of the IAB taxonomy above, if you are interested in the domain of Business and Finance, would it be sufficient to distinguish Business from Economy? Or, is it important to descend to more specific subcategories? For a human readable layout of a hierarchical model, it can be extremely helpful to lay out a taxonomy as a graph (such as above), or in a spreadsheet or similar tabular layout (which is how the IAB Tech Lab Taxonomy is distributed). For a machine-readable format, a tabular format is suitable, or consider a data format such as XML or JSON (the IAB Tech Lab distributes a machine-readable JSON version of their taxonomy as well).

At the extremes of being overly specific or overly general, your classification task becomes more difficult. We recommend experimenting with more general categories before creating many fine-grained distinctions. As you create categories, try to keep your categories mutually exclusive and be aware of categories that overlap.

If you find yourself in a situation where you have overlapping categories, consider adding a category to cover the overlap. Imagine you are building a model for classifying cooking recipes, with the categories Baking and Desserts which turn out to overlap considerably. In that case, consider splitting Desserts into Baked Desserts and Non-Baked Desserts. Alternatively, make Baked Desserts a subcategory of Baking. While intuition can be helpful, it is important to make these decisions based on the data instead of what you think the categories should look like. Sometimes the data will surprise you, and a model based on preconceptions rather than real data will suffer accordingly. Note that in the case of overlapping labels you may be interested in multi-label classification, where each document may be assigned to multiple categories.

If you discover mutually exclusive categories, you may be interested in creating multiple classifiers and combining their outputs. For instance, you may want one model to classify document topics, but another model to classify emotions expressed in documents. These are mutually exclusive because any given emotion can be expressed along with any given topic. Separating these categories with individual models may be the cleanest approach. It still may be worth experimenting to see if a single, multi-label model can be trained to classify documents across independent dimensions. This latter case is worth considering if you are limited in terms of time and resources, or you have specific linguistic features in mind that you hypothesize will be informative for one part of the classification task, and not harmful to the rest. Determining the correct approach is the “art” in model training, and requires designing and running experiments to inform the decision-making process.

A balanced dataset avoids creating a biased model. Each category should have roughly the same number of examples. Scrutinize the dataset's available metadata to ensure that the metadata values are evenly represented across different dimensions. For example, if you are drawing on multiple data sources, there shouldn’t be disproportionately more data from one source than another. If the data comes from different time periods, try to get a balanced distribution across time. Depending on the variables and the task, you may need to downsample your data in order to maintain balance, even if it reduces the size of your dataset. Don't hesitate to remove documents from your data-set in order to maintain balance as long as you have enough data. The overall size requirements differs from task to task; see the discussion of size below.

The training dataset must include examples of all the categories your model needs to classify. The data must also match the profile of the input data to be processed. If you are interested in classifying tweets, then you should collect tweets for your corpus (rather than, Wikipedia articles, instruction manuals, or sports commentary transcripts). Similarly, if the goal is classifying news articles by topic, your corpus should consist of news articles (not tweets or novels). Language (English, German), domain (financial, scientific, legal), genre (novel, sports news, tweets), register (formality of speech), and document size (tweets, blogs, magazine articles) are all aspects of the content that you should consider when assessing representativeness.

Remove "noisy documents" from the dataset, including documents that are not relevant for the task or do not contain sufficient lexical content. For example, a document which consists of a URL (and nothing else) is not a useful sample (unless you are analyzing URLs).

Note that you can use Rosette’s language identification functionality to select only documents in the target language. If you are interested in multiple languages, you will need to create a separate corpus for each language. Document classification models trained on one language will not function across other languages.

Generally, the more data you train with, the better your results. With more training data, a model can make more confident predictions. However, be careful not to pad your corpus with non-representative data only to increase the size of the corpus. Balance and representativeness are both important. One strategy to assess if you have enough data is to plot a learning curve. Since you cannot plot a learning curve until you get to the Test part of the MATTER cycle, when you are first collecting your dataset, collect more data than you think you need. As a rule of thumb, a minimal eval or development dataset would include hundreds of gold-standard documents per-category. Considering that these may only represent 10-20% of all the data you collect and use, you can’t collect too much data. You can always leave data unused, but it can be difficult to restart a data collection effort later on (or impossible if a data source becomes unavailable).

If you are using your data for a personal or academic project, there may be some intellectual property concerns, but there are greater issues when using a dataset for commercial purposes. Investigate who owns the data, and any licensing restrictions that exist on the data (if any). If you own the data, you may consider putting the data in the public domain, or creating an explicit license to allow or prevent others from deriving value from your data. You can find some licenses that are friendly for commercial purposes here. If you have concerns about intellectual property, always get legal advice.

Avoid duplicate documents or near-duplicate documents in your corpus. Duplicate data skews the frequencies of different vocabulary terms. Duplicates or near duplicates can also create "data leakage" if they occur across your training corpus -testing corpus partition. Data in your testing set should never overlap with your training set. There are various tools for finding duplicate documents in a corpus before you partition your dataset. We recommend:

Most plain-text documents are encoded as Unicode UTF-8. If you find data in other encodings, we recommend converting all your data to UTF-8. Here are some recommended tools for detecting and converting character encodings:

If the text in your documents includes markup tags (such as HTML, XML, Markdown, etc.), then you will need to extract the plain-text. Some suggested tools that automatically extract text from marked up documents are:

For documents in a particularly cumbersome markup language, consider converting it to a friendlier format using a tool such as Pandoc.

If you need to write your own software to extract text, start by researching what tools your go-to programming language offers; most programming languages will have libraries for parsing markup of various kinds. For example, Python’s standard library offers several tools for working with HTML, XML, and other similar markup languages. Some useful non-standard Python tools for working with various markup formats include:

Collected documents - especially from structured websites - may include metadata that is not part of the document body (i.e., titles, headers, footers, site map information). At best, metadata is noise, artificially over-representing the frequency of certain terms in your corpus. At worst, the metadata teaches your classifier to classify documents based on the metadata, rather than the document content. For example, if you collect documents from a website where the category label of the document is present in the metadata, then your classifier will learn from this signal since it will be a perfect predictor of the category.

On the website Quora, documents are tagged with categories. If you collected a corpus from Quora, you could use these tags in the metadata as if they were category annotations (which might save you from having to annotate the documents yourself). However, make sure that you don’t include these category tags in the document text! For example, here’s a Quora page about Blockchain technology that has been tagged:

If every document in your corpus that belongs to the "Blockchain Technology" category has the phrase "Blockchain Technology" in it, then your classifier is simply going to learn that any document with those terms belongs to the "Blockchain Technology" category. This result is called "overfitting" - your classifier ends up fitting exactly to the training data and it won’t generalize to real world data. If a document body happens to mention "block chain" (two words), or "cryptographically hashed, distributed transaction ledgers", these terms are relevant to the category "Blockchain Technology". But, if your classifier was trained explicitly on the category tags, and not the document body, these signals would be ignored. If your classifier is ever asked to classify a document that hasn’t been pre-tagged with Quora tags, it will not have that signal to rely on, so it will be at a major disadvantage. Additionally, if the documents in your test set have these tags, then the difficulty of classifying those documents will be artificially reduced to the point of triviality—all the classifier will need to do is emit the labels for the tags in the test documents. Your training will not be a simulation of a real-world classification problem (most documents in the real world will not have been annotated with category information like Quora), so your evaluation scores will be artificially high, overestimating the performance of your classifier. All this is to say, make sure that you take care to clean your documents and carefully consider what is actually part of the document body text - all document metadata should be separated in a preprocessing step.

Remember when selecting an annotation tool that for document categorization, annotators must assign labels (categories) to documents. Many annotation tools are intended for specific NLP tasks that require assigning labels to linguistic units (words, sentences, etc.) within documents rather than to whole documents. Make sure the annotation tool you choose has a suitable interface.

To keep track of a large collection of documents, we recommend assigning each document a Universally Unique Identifier (UUID) – if your documents don’t already have an identification scheme. A UUID will give you and your annotators a consistent, convenient way to refer to specific documents during the annotation process.

A popular open source annotation tool like brat is designed for tasks that require assigning labels to individual words, phrases, or other linguistic units within documents. For document categorization, however, it may be simpler to use a spreadsheet application, such as Google Sheets. Whatever tool you decide to use, keep in mind that it will be most convenient for your annotators if they can easily access the full text of the documents and choose a label from your model to assign to each document.

If there are inherent constraints on your category labels, best practice is to enforce those constraints in the annotation tool rather than rely on your annotators. For example, if you are using Google Sheets, set up data validation so that annotators’ inputs will be rejected if they fall outside your model’s defined set of categories. Any extra cognitive load or room for error that you add to the task has the potential to reduce the quality of the annotations (not to mention, making your annotators unhappy). Consider the size of the corpus, too. Some annotation environments may not be able to support a corpus beyond a certain size.

Finally, when compiling a project schedule, factor in the time needed to set up the annotation environment (software) for the annotators and the time to train them.

Here are some annotation tools for NLP applications:

When drawing up annotation guidelines, it is often helpful to run a pilot annotation effort with a small subset of annotators - who will also work on the broader annotation effort - or with just the author of the guidelines. The author(s) should do some annotations as a part of developing the guidelines. It’s very difficult to guide others through a task you have not performed yourself. The pilot annotation effort will reveal examples or edge cases that haven't been considered, and these will guide the authors in determining how to handle those cases (or to revise the model to account for them).

Annotation guidelines should describe each category and what it is intended to capture. While descriptions are useful, examples are essential. For each category, include positive examples—cases where the label in question applies. Negative examples - where the label in question does not apply - can be equally instructive. For examples that are not straightforward (such as those that annotators disagreed on), be sure to to discuss them and explain the reasoning that led to the final decision.

We measure annotator reliability because inconsistent annotation or lack of adherence to the annotation guidelines will lead to a less accurate classifier. Especially when starting a new project or on-boarding new human annotators, we check for reliable annotations by having a subset of data annotated in parallel by multiple human annotators.

To measure inter-annotator reliability, we recommend using Krippendorff’s alpha, a statistical inter-rater reliability metric. Krippendorff’s alpha scores range from -1.0 to 1.0 with 1.0 indicating perfect agreement between annotators. A score of 0.0 indicates agreement no better than random chance (as if your annotators picked their labels randomly out of a hat). A reliability score of 0.80 or greater is generally considered sufficient (though, the higher the better). Lower scores may indicate potential issues with your data, your annotation guidelines, or your annotators’ understanding of the task. A low level of inter-annotator agreement will ultimately lead to a less accurate classifier, so we recommend repeatedly measuring the reliability of your annotators until they achieve a satisfactory level of agreement. The cases where annotators disagree are usually good examples to include in your annotation guidelines. It can be useful to have a discussion about points of disagreement with your annotators as a group to reach a consensus.

The standard practice for supervised learning (where a statistical model learns from the gold-standard, human annotated training data) is to partition your annotated data into the following parts:

The motivation for splitting the data into training and testing sets is that if you evaluate a model on documents that it has seen during training, you are not measuring its true performance on new, unseen data that the model will encounter in the real world. As a quick check, you can evaluate a model against the data it was trained on. In this case, the results should be near perfect, and highly inflated compared to real-world performance.

Ideally, each of the training and testing partitions maintains a balanced distribution of categories. So, assuming your corpus was balanced to begin with, you could maintain balance by dividing the data labeled with each category into three parts and then putting each third into the training, development, and evaluation set.

For example, imagine a small corpus with 300 documents annotated with three food-related categories:

You might partition your corpus as follows:

The development cycle is as follows:

The process of trying different strategies to optimize your model (step #3) is called tuning—and specifically, you are tuning your model to perform well on the development set. Note, however, that you do not want to tune your classifier to the evaluation set.

When you are developing your corpus, it is not possible to know, a priori, how much data will be sufficient. Sometimes you will compile a corpus, perform careful manual annotation, and train a system only to discover that your classifier does not perform well even after tuning. One way to assess if your classifier is hampered by a lack of data is to plot a so-called learning curve.

Generally, the trend of this plot should show an increase in performance as more data is added. If this is not the case, it usually indicates a problem with the annotations. If the trend of this plot increases, but levels off after a certain amount of training data has been added, then adding additional data is unlikely to help.

If, however, the trend of the plot continues to increase without leveling off, we recommend adding additional data to your training set until you reach a point where adding more data achieves diminishing returns.

If your training set is imbalanced, the categories that have few examples relative to other classes may perform poorly. In this case, there are potentially two remedies:

Option 2 is only recommended if you have ruled out the possibility of having insufficient data.

The Train command is the only command needed to build models. It can also run cross-validation for a quick and simple way to validate a model once it has been created. Train has the following four subcommands:

The following parameters are used by the Train subcommends

$ bin/Train
usage: Train [options]... create lang config dataRootDir modelDir
       Train [options]... append dataRootDir modelDir
       Train [options]... train modelDir
       Train [options]... xval n-folds modelDir
usage: options
    -c,--cost <arg>              one or more (comma separated) cost params
                                 (C); default 0.01 - for train and xval. Only
                                 xval supports multiple cost values.      
   --negationWords <arg>         path to negation word list (create only)        
   --negativeLexicon <arg>       path to negative lexicon (create only)       
   --positiveLexicon <arg>       path to positive lexicon (create only)        
   --stop words <arg>             path to stop words list (create only)    
   --train                       train model after adding examples

TCatCLI runs the classifier that you trained with the Train command. It is a command-line version of the categories endpoint, allowing you to run the function using the trained model before integrating into Rosette Server.

$ bin/TCatCLI
 
usage: TCatCLI [options] file [...]
    --adm                             print out results in ADM JSON format
 -ct,--confidenceThreshold <arg>      show confidences above threshold
 -et,--elbowThresholding              use elbow thresholding (invalidates
                                      -st)
    --explanationSet <arg>            show top N positive features
                                      (requires -v)
 -f,--format <arg>                    (line|file|file-list), default=file
    --featureMatrixCategories <arg>   show matrix view of top N categories
 -m,--model <arg>                     model directory
    --maxResults <arg>                show top N categories, default 1
 -o,--output <arg>                    output file or directory. Output
                                      directory required for -f file-list
 -st,--scoreThreshold <arg>           show scores above threshold
 -v,--verbose                         verbose results

The command requires the -m (model) and -o (output) options, in addition to the input documents. The documents can be a line of text, a file, or a file-list. The default is file. If running with a line of text or file-list, the -f (format) option is required. TCatCLI should be run on documents not seen at training time.

When providing a file list to -f, the file list must contain one input file path per line. Additionally, your output argument (-o) must be a directory and not a single file, as the command will create one output file per input file in the list.  

MeasurementCLI generates the same statistics as Train xval, but instead of using cross-validation, it compares two lists of data, actual versus predicted. The first list is the gold-standard, annotated category each item belongs to, and the second list is the trained model's predicted category for each item. Each list contains one category per line. It is useful if you prepare your own train/test splits, as described in Evaluating on a Train/Test Partition.

$ bin/MeasurementCLI

 usage: MeasurementCLI [options]... actual predicted
  -m,--mode <arg>   (stats|matrix|list), default=stats, matrix supports up
                    to 26 categories

If you have a gold set of annotated documents in ADM format, you can score your model’s performance using Evaluate. By default, this evaluation will calculate per-category and overall (micro and macro) precision (P), recall (F), and F1 scores based on the model's performance in a single-label context. By specifying the --multilabel flag, the evaluation can also be performed in a multilabel context. In this case, the same P/R/F metrics will be calculated, as well as two additional metrics, hamming-loss and subset-loss. Note that by default, the multi-label evaluation will use a default raw score threshold of -0.25f. Only categories with a score greater than the default raw score threshold are returned. If you find that this value is suboptimal for your dataset and the not all expected categories are being returned, it can be updated by using the --score-threshold option. Additionally, the --include-categories and --exclude-categories options can be used to include or exclude specific categories from the evaluation.

$ bin/Evaluate

usage: Evaluate modelDir dataDir [options]
 -et,--elbow-thresholding        use elbow thresholding (only applicable 
                                 in multilabel mode)
 -i,--include-categories <arg>   categories to include in evaluation
 -m,--multilabel                 multilabel evaluation
 -mr,--max-results <arg>         max number of results to return
 -s,--score-threshold <arg>      raw score threshold (only applicable in
                                 multilabel mode)
 -t,--tokenize                   Ignore gold tokens

Consider the following pets dataset example with three categories: bird, cat, and dog. This is a toy dataset of Wikipedia documents for different animal breeds. It is too small for a real dataset, but is used here for demonstration purposes. You can find it in the distribution at classifier-field-training-kit-<version>/samples/data.

$ tree -d . 
. 
└── pets
    ├── bird 
    ├── cat 
    └── dog 

4 directories 
# sort documents by frequency per category 
$ find pets -name "*.txt" | cut -f2 -d'/' | sort | uniq -c | sort -nr  
  10 bird 
  10 cat 
  10 dog

The per-category statistics table displays the accuracy of your classifier for each category as well as averaged accuracy across all categories. The table columns are as follows:

The statistics are displayed for each category individually (per row) and there are additional average metrics at the foot of the table. These averages are explained as follows:

Per-category statistics:
                                   size     tp     fn     fp         P         R        F1
bird                                 10     10      0      0     1.000     1.000     1.000
cat                                  10      9      1      0     1.000     0.900     0.947
dog                                  10     10      0      1     0.909     1.000     0.952

Total counts:                        30     29      1      1         -         -         -
Macro Average:                        -      -      -      -     0.970     0.967     0.968
Micro Average:                        -      -      -      -     0.967     0.967     0.967

A confusion matrix is a representation of the true positives, false positives, and false negatives for a given evaluation run. In this case, the column headers of the matrix represent the category label that the classifier predicted, and the row labels (shown as the rightmost column) represent the gold standard category label that was annotated. The cells of the matrix contain the frequency of error types.

If all predictions were correct, all values would be 0 except on the diagonal.

In the following example, there are 10 total examples for cat. Nine were correctly identified, while One (column 3. row 2) was mis-predicted as dog ("c").

Confusion matrix:
      a       b       c <-- Predicted as
     10       0       0 |         10 a=bird
      0       9       1 |         10 b=cat
      0       0      10 |         10 c=dog

A confusion list is a representation of the false positives and false negatives for a given evaluation run. It contains essentially the same information as the confusion matrix, however, the true positives are omitted (these are the values that occur along the diagonal in the confusion matrix). This view of the statistics can help to suss out what are the most common error types and which categories the classifier is confusing most often. It is also useful when you have so many categories that a confusion matrix becomes unwieldy (e.g., more than 10 categories results in a larger than 10 by 10 matrix which can make it difficult to read).

Confusion list:
actual                            predicted               errors
cat                               dog                          1

Note the misspelling of "convenient" as "convienient". Misspellings are intentionally included as they may be sparser than correct spellings, so a finite training corpus may lack examples of the misspellings ("convenient" with the correct spelling is also included in the lexicon). Also note the British spelling variation of "enthrall" as "enthral" ("enthrall" with two "l"s is also included in the lexicon). These lexicons were developed for the purpose of analyzing sentiment in informal, social media communications, specifically tweets. As such, common misspellings or variations are helpful to capture along with correct spellings. If misspellings are excluded from your lexicons, their signals may be lost to the model, and may reduce the model’s ability to learn.

When developing sentiment lexicon resources, it is important to consider an inherent property of natural language: lexical ambiguity. Many words are polysemous (have multiple meanings), and it is through context, or world knowledge, that humans interpret the intended meaning. As such, it is important to avoid, as much as possible, words that are relatively ambiguous across different sentiment categories.