Metrics¶

Any keras supported Metric class can be used with ml4ir. ml4ir comes prepackaged with the following popular search metrics.

MeanReciprocalRank¶

class ml4ir.applications.ranking.model.metrics.metrics_impl.MRR(name='mean', dtype=None)¶

Bases: ml4ir.applications.ranking.model.metrics.metrics_impl.MeanRankMetric

Custom metric class to compute the Mean Reciprocal Rank.

Calculates the mean of the reciprocal ranks of the clicked records from a list of queries.

Examples

>>> `y_true` is [[0, 0, 1], [0, 1, 0]]
>>> `y_pred` is [[0.1, 0.9, 0.8], [0.05, 0.95, 0]]
>>> then the MRR is 0.75

AverageClickRank¶

class ml4ir.applications.ranking.model.metrics.metrics_impl.ACR(name='mean', dtype=None)¶

Bases: ml4ir.applications.ranking.model.metrics.metrics_impl.MeanRankMetric

Custom metric class to compute the Average Click Rank.

Calculates the mean of the ranks of the clicked records from a list of queries.

Examples

>>> `y_true` is [[0, 0, 1], [0, 1, 0]]
>>> `y_pred` is [[0.1, 0.9, 0.8], [0.05, 0.95, 0]]
>>> then the ACR is 1.50

CategoricalAccuracy¶

Top5CategoricalAccuracy¶

class ml4ir.applications.classification.model.metrics.metrics_impl.Top5CategoricalAccuracy(name='top_5_categorical_accuracy', **kwargs)¶

Bases: keras.metrics.metrics.TopKCategoricalAccuracy

Custom metric class to compute the Top K Categorical Accuracy.

Currently a wrapper around tf.keras.metrics.TopKCategoricalAccuracy that squeezes one dimension. It maintains consistency of arguments to __init__

Creates a CategoricalAccuracy instance

Parameters:	name (str) – Name of the metric

update_state(y_true, y_pred, sample_weight=None)¶

Squeeze the second dimension(axis=1) and compute top K categorical accuracy

Parameters:	y_true (Tensor object) – Tensor containing true class labels Shape : [batch_size, 1, num_classes] y_pred (Tensor object) – Tensor containing predicted scores for the classes Shape : [batch_size, 1, num_classes] sample_weight (dict) – Dictionary containing weights for the classes to measure weighted average metric
Returns:	Top K categorical accuracy computed on y_true and y_pred
Return type:	Tensor object

Notes

Input shape is a 3 dimensional tensor of size (batch_size, 1, num_classes). We are squeezing the second dimension to follow the API of tf.keras.metrics.TopKCategoricalAccuracy

Axis 1 of y_true and y_pred must be of size 1, otherwise tf.squeeze will throw error.