Blog

Understanding text classification with machine learning

Text classification assigns each document to one of a set of predefined categories. A product review can be labeled as good or bad, a short message by the emotion it expresses, and a support ticket by its topic or urgency.

Classical approaches represented each document as a bag of word counts and discarded the order of the words. Modern neural networks learn their own representation of the text instead: they turn each word into a vector, let the words of a document interact through attention, and summarize the result to predict a class.

This post explains how text classification works in Neural Designer, from preparing the data set to deploying the model. It covers tokenization, the embedding, multi-head attention, and pooling layers, training, testing analysis, and deployment.

What is text classification?
From bag of words to learned representations
Preparing the data set
Tokenization and vocabulary
Neural network architecture
Embedding layer
Multi-head attention layer
Pooling and dense layers
Training strategy
Testing analysis
Model deployment
Examples
Conclusions

What is text classification?

A text classification model receives a piece of text, such as a sentence, a review, or an email, and returns the class it belongs to, together with a probability for each class.

Depending on the number of classes, there are two types of problems:

Binary classification separates two classes, for example, spam and legitimate email, or positive and negative reviews.
Multiple classification assigns each document to one of three or more classes, for example, the emotion expressed in a message or the department that should handle a support ticket.

In both cases, each document belongs to exactly one class. Common applications include sentiment analysis, emotion detection, spam filtering, ticket routing, and document categorization.

Text classification is a supervised learning task. The model learns from documents that have already been labeled, often by experts in the field, and then applies what it has learned to new documents.

From bag of words to learned representations

For many years, the standard approach was the bag of words. Documents were first normalized: they were converted to lowercase and stripped of punctuation and stop words, and the remaining words were reduced to their stems. Then, each document was represented by a vector that counted how many times each dictionary word appeared in it.

That representation has well-known limitations:

It ignores word order. “The battery is not good, it is bad” and “the battery is not bad, it is good” contain the same words, so they get the same vector.
The vectors are as long as the vocabulary and almost entirely zeros, so memory grows with the size of the corpus.
All words are equally unrelated to each other: “excellent” is as different from “great” as it is from “terrible”.
Normalization can remove useful information. “Not” appears in many stop-word lists, yet it reverses the meaning of a review.

Neural Designer takes a different approach. It keeps the words in their original order and learns the representation directly from the data. An embedding layer maps each word to a dense vector, an attention layer relates each word to the others in the same document, and training determines which words and combinations matter for the classification. As a result, you don’t need to remove stop words, stem words, or build word-count tables before training.

Preparing the data set

To create a text classification model, select Text classification on the Neural Designer start page and import a data file. The file must be a .csv or .txt file with one document per line. The last field of each line is the class label, and the text comes before it.

Neural Designer supports tab and semicolon separators. The following lines show the format of the Amazon reviews example, in which a tab separates each review from its label:

Good case, Excellent value.	Good
The mic is great.	Good
Needless to say, I wasted my money.	Bad
What a waste of money and time!.	Bad

The import wizard also detects an optional header line and an optional first column with sample identifiers. A file can contain several text columns, for example, the title and the body of a ticket. In that case, Neural Designer inserts a marker token at the start of each column, so the model can tell which part of the document each word comes from.

The number of different labels determines the type of problem. With two labels, the model has a single output that gives the probability of the positive class. With three or more, it has one output per class. Labels are not case-sensitive, so “Good” and “good” are the same class.

By default, Neural Designer splits the documents at random into training (60%), validation (20%), and testing (20%) subsets. The Dataset tab shows the vocabulary size, the maximum sequence length, the number of target classes, and the number of samples.

Tokenization and vocabulary

A neural network works with numbers, so the first step is to convert each document into a sequence of integers. Neural Designer does this with a word-level tokenizer:

The text is converted to lowercase.
The text is split into words, that is, into runs of letters and digits.
Each punctuation mark becomes a token of its own. Marks such as “!” or “?” often carry sentiment, so they are kept rather than removed.
Each token is replaced by its index in the vocabulary.

The vocabulary is built from the words of the data set, ranked by frequency, and holds up to 20,000 entries. It also contains four reserved tokens:

[PAD] fills the positions after the end of shorter documents.
[UNK] replaces words that are not in the vocabulary.
[START] and [END] mark the beginning and the end of each document.

All sequences have the same length so that the network can process them in batches. By default, this length is that of the longest document plus the start and end tokens. Shorter documents are padded at the end with [PAD] tokens. For example, with a sequence length of 10, the review “Good case, Excellent value.” becomes the following sequence:

Position	1	2	3	4	5	6	7	8	9	10
Token	[START]	good	case	,	excellent	value	.	[END]	[PAD]	[PAD]

Two settings in the Dataset tab control this step:

Sequence length limit truncates longer documents. By default, there is no limit. If the longest document has more than 256 tokens, Neural Designer offers to truncate the sequences to 256 tokens, which saves memory and training time when only a few documents are very long.
Minimum word frequency leaves out of the vocabulary the words that appear fewer times than this value; they are then encoded as [UNK]. Raising it reduces the vocabulary size and, therefore, the number of parameters in the embedding layer.

The Calculate vocabulary statistics task helps you choose these values. It reports the vocabulary size, the number of words that appear only once, the percentage of unknown words in each subset, the median, 90th-percentile, and 99th-percentile document lengths, the percentage of padding, and the vocabulary coverage curve.

Neural network architecture

When you import a text data set, Neural Designer builds a text classification network with the following layers:

TokenizerConverts the text into a sequence of vocabulary indices.L

Embedding with positional encodingTurns each token into a vector and adds its position in the document.L × d

Multi-head self-attentionRelates each word to every other word in the document.L × d

Average poolingAverages the word vectors into one document vector, ignoring padding.d

Dense layer (ReLU)Combines the features of the document.h

Dense layer (sigmoid or softmax)Gives the probability of each class.1 or K

Layers of a text classification network in Neural Designer. $L$ is the sequence length, $d$ the embedding dimension, $h$ the number of neurons in the hidden dense layer, and $K$ the number of classes. Highlighted layers have trainable parameters.

The embedding, attention, and dense layers contain the trainable parameters. The tokenizer and the pooling layer have none: the first converts text into indices, and the second summarizes the sequence.

This architecture is an encoder based on the attention mechanism of the Transformer [1]. It is compact, and it captures relationships between words that a bag of words can’t represent. The following sections describe its main layers.

Embedding layer

The embedding layer turns each token index into a dense vector of $d$ real numbers, where $d$ is the embedding dimension. It stores a matrix $\mathbf{E}$ with one row per vocabulary entry and $d$ columns. The vector of a token is the corresponding row of that matrix.

The values of $\mathbf{E}$ are trainable parameters. They start at random values, and training adjusts them so that words that play a similar role in the classification end up with similar vectors.

An embedding vector does not contain information about where a word appears in the document. For this reason, Neural Designer adds a sinusoidal positional encoding $\mathbf{p}_i$ to the vector of the word at position $i$:

$$ \mathbf{x}_i=\sqrt{d}\,\mathbf{E}_{t_i}+\mathbf{p}_i, $$

where $t_i$ is the vocabulary index of the token at position $i$. The components of the positional encoding are

$$ p_{i,j}= \begin{cases} \sin\left(i/10000^{2j/d}\right), & j<d/2,\\ \cos\left(i/10000^{2(j-d/2)/d}\right), & j\ge d/2. \end{cases} $$

The scaling factor $\sqrt{d}$ keeps the word vectors and the positional encoding at comparable magnitudes. Padding tokens are mapped to zero vectors, so they don’t contribute to the following layers.

The output of the embedding layer is a matrix with one row per position and $d$ columns. The embedding dimension is 64 by default, and you can change it in the Neural network tab. Larger values can represent finer distinctions between words, but they add parameters: the embedding layer alone has $V\times d$ parameters, where $V$ is the vocabulary size.

Multi-head attention layer

After the embedding layer, each word is still represented on its own, but the meaning of a word depends on its context. “Good” contributes opposite sentiments in “very good” and in “not good”. The attention layer lets each position gather information from all the other positions in the same document.

For each position, the layer computes a query, a key, and a value vector by multiplying the input by three trainable matrices. The attention weight between two positions measures how well the query of one matches the key of the other. Each output vector is then a weighted average of the value vectors:

$$ \operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})= \operatorname{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^{\mathsf T}}{\sqrt{d_k}}\right)\mathbf{V}, $$

where $d_k$ is the dimension of the key vectors.

A multi-head attention layer runs several attention operations, or heads, in parallel. With $H$ heads, each head works with vectors of dimension $d_k=d/H$, so different heads can focus on different relationships between words. The outputs of all the heads are concatenated and multiplied by a fourth trainable matrix, which gives a vector of dimension $d$ for each position.

In text classification, attention is bidirectional: each word can attend to the words both before and after it. Padding positions are masked, so they receive no attention.

In the Neural network tab, you can set the number of heads, which must divide the embedding dimension, and you can add more attention layers.

Pooling and dense layers

The attention layer still produces one vector per position. To classify the whole document, the pooling layer summarizes the sequence into a single vector of dimension $d$. By default, Neural Designer uses average pooling:

$$ \mathbf{z}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{y}_i, $$

where $\mathbf{y}_i$ is the output of the attention layer at position $i$ and $n$ is the number of positions that are not padding. Because padding is excluded, short and long documents are treated alike. You can also select max pooling, which keeps the largest value of each feature over the document.

The dense layers then map the document vector to the classes:

A hidden dense layer with a rectified linear (ReLU) activation function. By default, it has 64 neurons.
An output dense layer. For binary classification, it has a single neuron with a logistic (sigmoid) activation function, which gives the probability of the positive class. For multiple classification, it has one neuron per class with a softmax activation function, which gives probabilities for all the classes that add up to one.

The predicted class is the one with the highest probability. In binary classification, a probability above 0.5 means the positive class.

Training strategy

Neural Designer configures the training strategy for text classification automatically:

Loss index. The error term is the cross-entropy, binary for two classes and categorical for more classes. By default, there is no regularization, but you can add L1 or L2 regularization.
Optimization algorithm. Adaptive moment estimation (Adam) with a learning rate of 0.001, batches of 64 documents, and a maximum of 75 epochs. Stochastic gradient descent is also available.

Quasi-Newton and Levenberg-Marquardt are not offered for text models. These methods store matrices whose size grows with the square of the number of parameters, and text models have many parameters in their embedding and attention layers. Mini-batch methods such as Adam scale to large vocabularies and data sets. You can read more about this in our post on neural network optimizers.

During training, Neural Designer plots the training and validation errors for each epoch. The validation error shows when the model starts to overfit, that is, when it memorizes the training documents instead of learning patterns that generalize to new ones.

Text data sets are read from disk in batches during training, so they don’t need to fit in memory. On computers with an NVIDIA GPU of the Ampere generation or newer, training can also run on the GPU.

Testing analysis

After training, the testing analysis evaluates the model on the testing documents, which were not used during training. Neural Designer offers the following tasks for text classification models.

Confusion matrix

The confusion matrix compares the actual and predicted classes. Rows represent the actual classes, and columns represent the predicted classes, so correct predictions lie on the diagonal. The matrix shows which classes the model confuses with each other. For example, this is the confusion matrix of the Amazon reviews example on its 200 testing documents:

Actual / predicted	bad	good
bad	68	30
good	35	67

Classification metrics

Neural Designer calculates the classification metrics from the confusion matrix. For a class $c$, let $TP$, $FP$, and $FN$ be the numbers of true positives, false positives, and false negatives for that class. Then

$$ \text{precision}=\frac{TP}{TP+FP},\qquad \text{recall}=\frac{TP}{TP+FN},\qquad F_1=\frac{2\cdot\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}}. $$

For binary models, the binary classification tests report accuracy, error rate, sensitivity, specificity, precision, the F1 score, the Matthews correlation coefficient, and other indicators. For multiple classification models, Neural Designer reports precision, recall, and F1 score for each class, together with their macro and weighted averages. The macro average weights all classes equally, which matters when some classes are rare.

ROC curve, cumulative gain, and lift chart

For binary models, the ROC curve plots the sensitivity against one minus the specificity for every decision threshold. Neural Designer calculates the area under the curve and the optimal threshold. The cumulative gain and lift charts show how well the model ranks the documents of the positive class. You can read more about these methods in our post on testing methods for binary classification models.

Misclassified samples

This task lists the testing documents that the model classified incorrectly, with their original text, actual class, and predicted class. Reading them is often the quickest way to understand the errors of a model, which typically involve negations, sarcasm, or ambiguous labels.

What counts as a good result depends on the task: 90% accuracy can be excellent for one problem and poor for another. Compare the results with a simple baseline, such as always predicting the most frequent class, and with published results for similar tasks.

Model deployment

Once the model has been tested, you can use it to classify new documents in three ways:

Calculate outputs. Type a sentence, or one text per column for models with several text columns, and Neural Designer shows the probability of each class and the predicted class.
Export output data. Classify a whole file of new documents. Neural Designer writes a CSV file with the predicted class and the probability of each class for every document.
Export deployment package. Create a folder with the trained model, the Neural Designer engine, and a Python or C program that calls it. You can then classify text from your own applications without opening Neural Designer. The package runs on 64-bit Windows.

For example, the Python program of the Amazon reviews package can be used as follows:

python amazon_reviews.py "The battery lasts all day."
python amazon_reviews.py --probs "The battery lasts all day."
python amazon_reviews.py --file reviews.txt

The vocabulary and the sequence length are saved with the model, so new documents are encoded exactly like the training documents. See the model deployment tutorial for more details.

Examples

The following examples, available on the Neural Designer start page, show the complete workflow with real data:

Amazon review sentiment classification: binary classification of 1,000 product reviews as good or bad [2]. The model reaches 67.5% accuracy on 200 testing documents.
Emotion classification from text: multiple classification of 20,000 short texts into six emotions: joy, sadness, anger, fear, love, and surprise [3]. The model reaches 86.33% accuracy and a macro F1 score of 0.832 on 4,000 testing documents.

Conclusions

Text classification assigns documents to predefined classes, and it is one of the most common applications of machine learning to language.

Neural Designer implements a modern neural approach. A word-level tokenizer converts each document into a sequence of tokens. An embedding layer with positional encoding turns the tokens into vectors, and a multi-head attention layer relates each word to its context. Then, a pooling layer and dense layers produce the class probabilities.

The whole workflow, from importing the data to testing and deploying the model, runs in a graphical interface without writing code.

References

Vaswani, A. et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems 30.
Kotzias, D., Denil, M., de Freitas, N. and Smyth, P. (2015). From group to individual labels using deep features. Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J. and Chen, Y.-S. (2018). CARER: Contextualized affect representations for emotion recognition. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.