Metrics

Any keras supported Metric class can be used with ml4ir. ml4ir comes prepackaged with the following popular search metrics.

MeanReciprocalRank

class ml4ir.applications.ranking.model.metrics.metrics_impl.MRR(feature_config: ml4ir.base.features.feature_config.FeatureConfig, metadata_features: Dict[KT, VT], name='MRR', state='new', **kwargs)

Bases: ml4ir.applications.ranking.model.metrics.metrics_impl.MeanRankMetric

Custom metric class to compute the Mean Reciprocal Rank.

Calculates the mean of the reciprocal ranks of the clicked records from a list of queries.

Examples

>>> `y_true` is [[0, 0, 1], [0, 1, 0]]
>>> `y_pred` is [[0.1, 0.9, 0.8], [0.05, 0.95, 0]]
>>> `mask` is [[1, 1, 1], [1, 1, 1]] and
>>> `rank` is [[1, 3, 2], [3, 1, 2]]
>>> then the MRR is 0.75

Creates a MRR instance to compute mean of reciprocal rank

Parameters:
  • feature_config (FeatureConfig object) – FeatureConfig object that defines the configuration for each model feature
  • metadata_features (dict) – Dictionary of metadata feature tensors that can be used to compute custom metrics
  • name (str) – Name of the metric
  • state ({"new", "old"}) – State of the metric

AverageClickRank

class ml4ir.applications.ranking.model.metrics.metrics_impl.ACR(feature_config: ml4ir.base.features.feature_config.FeatureConfig, metadata_features: Dict[KT, VT], name='ACR', state='new', **kwargs)

Bases: ml4ir.applications.ranking.model.metrics.metrics_impl.MeanRankMetric

Custom metric class to compute the Average Click Rank.

Calculates the mean of the ranks of the clicked records from a list of queries.

Examples

>>> `y_true` is [[0, 0, 1], [0, 1, 0]]
>>> `y_pred` is [[0.1, 0.9, 0.8], [0.05, 0.95, 0]]
>>> `mask` is [[1, 1, 1], [1, 1, 1]] and
>>> `rank` is [[1, 3, 2], [3, 1, 2]]
>>> then the ACR is 1.50

Creates a ACR instance to compute mean of rank

Parameters:
  • feature_config (FeatureConfig object) – FeatureConfig object that defines the configuration for each model feature
  • metadata_features (dict) – Dictionary of metadata feature tensors that can be used to compute custom metrics
  • name (str) – Name of the metric
  • state ({"new", "old"}) – State of the metric

CategoricalAccuracy

class ml4ir.applications.classification.model.metrics.metrics_impl.CategoricalAccuracy(feature_config: ml4ir.base.features.feature_config.FeatureConfig, metadata_features: Dict[KT, VT], name='categorical_accuracy', state='new', **kwargs)

Bases: tensorflow.python.keras.metrics.CategoricalAccuracy

Custom metric class to compute the Categorical Accuracy.

Currently just a wrapper around tf.keras.metrics.CategoricalAccuracy to maintain consistency of arguments to __init__

Creates a CategoricalAccuracy instance

Parameters:
  • feature_config (FeatureConfig object) – FeatureConfig object that defines the configuration for each model feature
  • metadata_features (dict) – Dictionary of metadata feature tensors that can be used to compute custom metrics
  • name (str) – Name of the metric
  • state ({"new", "old"}) – State of the metric

Top5CategoricalAccuracy

class ml4ir.applications.classification.model.metrics.metrics_impl.Top5CategoricalAccuracy(feature_config: Optional[ml4ir.base.features.feature_config.FeatureConfig] = None, metadata_features: Dict[KT, VT] = {}, name='top_k_categorical_accuracy', state='new', **kwargs)

Bases: tensorflow.python.keras.metrics.TopKCategoricalAccuracy

Custom metric class to compute the Top K Categorical Accuracy.

Currently a wrapper around tf.keras.metrics.TopKCategoricalAccuracy that squeezes one dimension. It maintains consistency of arguments to __init__

Creates a CategoricalAccuracy instance

Parameters:
  • feature_config (FeatureConfig object) – FeatureConfig object that defines the configuration for each model feature
  • metadata_features (dict) – Dictionary of metadata feature tensors that can be used to compute custom metrics
  • name (str) – Name of the metric
  • state ({"new", "old"}) – State of the metric
update_state(y_true, y_pred, sample_weight=None)

Squeeze the second dimension(axis=1) and compute top K categorical accuracy

Parameters:
  • y_true (Tensor object) – Tensor containing true class labels Shape : [batch_size, 1, num_classes]
  • y_pred (Tensor object) – Tensor containing predicted scores for the classes Shape : [batch_size, 1, num_classes]
  • sample_weight (dict) – Dictionary containing weights for the classes to measure weighted average metric
Returns:

Top K categorical accuracy computed on y_true and y_pred
Return type:

Tensor object

Notes

Input shape is a 3 dimensional tensor of size (batch_size, 1, num_classes). We are squeezing the second dimension to follow the API of tf.keras.metrics.TopKCategoricalAccuracy

Axis 1 of y_true and y_pred must be of size 1, otherwise tf.squeeze will throw error.