Calculating BCubed metrics for clustering tasks

Introduction

The B-Cubed algorithm was first introduced by Bagga, A. and Baldwin B. (1998) in their paper on Entity-Based Cross-Document Coreferencing Using the Vector Space Model. The algorithm compares a predicted clustering with a ground truth (or gold standard) clustering through element-wise precision and recall scores. BCubed metrics provide a robust evaluation method for comparing predicted clusters with the ground truth, particularly in cases of overlapping or soft clusters.

For each element, the predicted and ground truth clusters containing the element are compared, and then the mean over all elements is taken. The B-Cubed algorithm can be useful in unsupervised techniques where the cluster labels are not available, because unlike macro-averaged metrics, it focuses on element-wise operations.

From the paper, two simple equations were devised calculating precision and recall scores for the predicted clustering:

Precision = \frac{1}{\sum elements} \sum_{i = 1}^{n} \frac{{(count of element)}^{2}}{count of all elements in cluster}

Recall = \frac{1}{\sum elements} \sum_{i = 1}^{n} \frac{{(count of element)}^{2}}{count of total elements from this category}

F-score = \frac{1}{k} \sum_{i = 1}^{n} \frac{2 \times {Precision (C)}_{k} \times {Recall (C)}_{k}}{{Precision (C)}_{k} + {Recall (C)}_{k}}

where n above denotes the number of categories in the cluster and k is the number of predicted clusters. Precision(C)_k and Recall(C)_k are the ‘partial’ precision and recalls for each cluster.

Visual Example

The image below consists of four clusters as predicted from a model, where each data point is assigned a colour representing a “label” or “category”: Blue, Red and Green.

Four predicted clusters drawn as circles. Each mixes blue, red and green points: cluster 1 has 4 blue, 2 red and 1 green; cluster 2 has 2 blue, 2 red and 3 green; cluster 3 has 1 blue and 5 red; cluster 4 has 1 blue, 2 red and 3 green. — Predicted Clustering for 3 Labels (Blue, Red, Green)

The correct clustering however would result in three clusters, instead of four, representing the amount of labels. This is called ground truth clustering. The “Blue” cluster would contain 8 data points, the “Red” would contain 11 and the “Green” would contain 7, summing up to 26 data points in total.

Three ground truth clusters drawn as circles: one of blue points, one of green points and one of red points. — Ground Truth Clustering for 3 Labels (Blue, Red, Green)

The tables below show the element-wise precision, recall, and f-score, which are calculated based on the provided formulas and focusing on the summation part to show the element-wise operations. For precision, the denominator equals the total elements in the predicted cluster, while for recall the denominator is the total elements for that label (also equal to the ground truth cluster for that element).

Table 1: Precision
Cluster / Label	Blue	Red	Green
Cluster 1	4²/7 = 2.286	2²/7 = 0.571	1²/7 = 0.143
Cluster 2	2²/7 = 0.571	2²/7 = 0.571	3²/7 = 1.286
Cluster 3	1²/6 = 0.167	5²/6 = 4.167	0²/6 = 0
Cluster 4	1²/6 = 0.167	2²/6 = 0.667	3²/6 = 1.500

Table 2: Recall
Cluster / Label	Blue	Red	Green
Cluster 1	4²/8 = 2.000	2²/11 = 0.364	1²/7 = 0.143
Cluster 2	2²/8 = 0.500	2²/11 = 0.364	3²/7 = 1.286
Cluster 3	1²/8 = 0.125	5²/11 = 2.273	0²/7 = 0
Cluster 4	1²/8 = 0.125	2²/11 = 0.364	3²/7 = 1.286

Table 3: F-score
Cluster / Label	Blue	Red	Green
Cluster 1	0.533	0.222	0.143
Cluster 2	0.267	0.222	0.429
Cluster 3	0.143	0.588	0
Cluster 4	0.143	0.235	0.462

Element-wise metrics: Precision (Table 1), Recall (Table 2), F-score (Table 3)

By summing the above values in each table for each cluster separately, and following the formulas, the BCubed metrics are:

Metric	Cluster 1	Cluster 2	Cluster 3	Cluster 4
Partial Precision	4²/7 + 2²/7 + 1²/7 = 3	3²/7 + 2²/7 + 2²/7 = 2.428	1²/6 + 0 + 5²/6 = 4.333	1²/6 + 3²/6 + 2²/6 = 2.333
Partial Recall	4²/8 + 1²/7 + 2²/11 = 2.506	2²/8 + 3²/7 + 2²/11 = 2.149	1²/8 + 0 + 5²/11 = 2.398	1²/8 + 3²/7 + 2²/11 = 1.774
Partial F-score	2.73	2.279	3.088	2.015

Partial Precision, Recall and F-score for cluster classes calculated with the BCubed method

BCubed metrics	Value
Precision	0.465 / 46.5%
Recall	0.339 / 33.9%
F-score	0.392 / 39.2%

Final BCubed metrics

BCubed Python Package

The bcubed-metrics package can help calculate the metrics only by providing 2 dictionaries; the predicted clustering and ground truth clustering.

You can find the documentation here along with a usage example. To install the package simply use pip:

pip install bcubed-metrics