Metrics

Module of metrics that measure the quality of detection results against true anomalies.

adtk.metrics.f1_score(y_true, y_pred, recall_thresh=0.5, precision_thresh=0.5)

F1 score of prediction.

F1 score is the harmonic mean of precision and recall. For more details about precision and recall, please refer to function precision and recall.

Parameters

• y_true (pandas Series or DataFrame, list, or dict) –

  Labels or lists of true anomalies.

  • If pandas Series, it is treated as binary labels along time index.

  • If pandas DataFrame, each column is a binary series and is treated as an independent type of anomaly.

  • If list, a list of events where an event is a pandas Timestamp if it is instantaneous or a 2-tuple of pandas Timestamps if it is a closed time interval.

  • If dict, each key-value pair is a list of events and is treated as an independent type of anomaly.

• y_pred (pandas Series or DataFrame, list, or dict) –

  Labels or lists of predicted anomalies.

  • If pandas Series, it is treated as binary labels along time index.

  • If pandas DataFrame, each column is a binary series and is treated as an independent type of anomaly.

  • If list, a list of events where an event is a pandas Timestamp if it is instantaneous or a 2-tuple of pandas Timestamps if it is a closed time interval.

  • If dict, each key-value pair is a list of events and is treated as an independent type of anomaly.

• recall_thresh (float, optional) – Threshold of recall calculation. Only used if input is list or dict. For more details, please refer to function recall. Default: 0.5.

• precision_thresh (float, optional) – Threshold of precision calculation. Only used if input is list or dict. For more details, please refer to function precision. Default: 0.5.

Returns

Score(s) for each type of anomaly.

Return type

float or dict

adtk.metrics.iou(y_true, y_pred)

IoU (Intersection over Union) score of prediction.

Intersection over union is the length ratio between time segments that are identified as anomalous in both lists and those identified by at least one of the two lists.

When the input is anomaly labels, metric calculation counts the number of anomalous time points.

When the input is lists of anomalous time windows, metric calculation measure the length of time segments.

Parameters

• y_true (pandas Series or DataFrame, list, or dict) –

  Labels or lists of true anomalies.

  • If pandas Series, it is treated as binary labels along time index.

  • If pandas DataFrame, each column is a binary series and is treated as an independent type of anomaly.

  • If list, a list of events where an event is a pandas Timestamp if it is instantaneous or a 2-tuple of pandas Timestamps if it is a closed time interval.

  • If dict, each key-value pair is a list of events and is treated as an independent type of anomaly.

• y_pred (pandas Series or DataFrame, list, or dict) –

  Labels or lists of predicted anomalies.

  • If pandas Series, it is treated as binary labels along time index.

  • If pandas DataFrame, each column is a binary series and is treated as an independent type of anomaly.

  • If list, a list of events where an event is a pandas Timestamp if it is instantaneous or a 2-tuple of pandas Timestamps if it is a closed time interval.

  • If dict, each key-value pair is a list of events and is treated as an independent type of anomaly.

Returns

Score(s) for each type of anomaly.

Return type

float or dict

adtk.metrics.precision(y_true, y_pred, thresh=0.5)

Precision score of prediction.

Precision, a.k.a. positive predictive value (PPV), is the percentage of predicted anomalies that are true anomalies.

When the input is anomaly labels, metric calculation treats every time point with positive label as an independent event. An anomalous time point that are detected as anomalous is considered as a successful detection.

When the input is lists of anomalous time windows, metric calculation treats every anomalous time window as an independent event. A detected event is considered as a successfully detection if the percentage of this time window covered by the true anomaly list is greater or equal to thresh. Note that input time windows will be merged first if overlapped time windows exists in the list.

Parameters

• y_true (pandas Series or DataFrame, list, or dict) –

  Labels or lists of true anomalies.

  • If pandas Series, it is treated as binary labels along time index.

  • If pandas DataFrame, each column is a binary series and is treated as an independent type of anomaly.

  • If list, a list of events where an event is a pandas Timestamp if it is instantaneous or a 2-tuple of pandas Timestamps if it is a closed time interval.

  • If dict, each key-value pair is a list of events and is treated as an independent type of anomaly.

• y_pred (pandas Series or DataFrame, list, or dict) –

  Labels or lists of predicted anomalies.

  • If pandas Series, it is treated as binary labels along time index.

  • If pandas DataFrame, each column is a binary series and is treated as an independent type of anomaly.

  • If list, a list of events where an event is a pandas Timestamp if it is instantaneous or a 2-tuple of pandas Timestamps if it is a closed time interval.

  • If dict, each key-value pair is a list of events and is treated as an independent type of anomaly.

• thresh (float, optional) – Threshold of a hit. Only used if input is list or dict. Default: 0.5.

Returns

Score(s) for each type of anomaly.

Return type

float or dict

adtk.metrics.recall(y_true, y_pred, thresh=0.5)

Recall score of prediction.

Recall, a.k.a. sensitivity, hit rate, or true positive rate (TPR), is the percentage of true anomalies that are detected successfully.

When the input is anomaly labels, metric calculation treats every time point with positive label as an independent event. An anomalous time point that are detected as anomalous is considered as a successful detection.

When the input is lists of anomalous time windows, metric calculation treats every anomalous time window as an independent event. A true event is considered as successfully detected if the percentage of this time window covered by the detected list is greater or equal to thresh. Note that input time windows will be merged first if overlapped time windows exists in the list.

Parameters

• y_true (pandas Series or DataFrame, list, or dict) –

  Labels or lists of true anomalies.

  • If pandas Series, it is treated as binary labels along time index.

  • If pandas DataFrame, each column is a binary series and is treated as an independent type of anomaly.

  • If list, a list of events where an event is a pandas Timestamp if it is instantaneous or a 2-tuple of pandas Timestamps if it is a closed time interval.

  • If dict, each key-value pair is a list of events and is treated as an independent type of anomaly.

• y_pred (pandas Series or DataFrame, list, or dict) –

  Labels or lists of predicted anomalies.

  • If pandas Series, it is treated as binary labels along time index.

  • If pandas DataFrame, each column is a binary series and is treated as an independent type of anomaly.

  • If list, a list of events where an event is a pandas Timestamp if it is instantaneous or a 2-tuple of pandas Timestamps if it is a closed time interval.

  • If dict, each key-value pair is a list of events and is treated as an independent type of anomaly.

• thresh (float, optional) – Threshold of a hit. Only used if input is list or dict. Default: 0.5.

Returns

Score(s) for each type of anomaly.

Return type

float or dict