6.1.5.5. hat.metrics¶

Metrics widely used for different datasets in HAT.

6.1.5.5.1. Metrics¶

`Accuracy`	Computes accuracy classification score.
`AccuracySeg`	# TODO(min.du, 0.5): merged with Accuracy #.
`TopKAccuracy`	Computes top k predictions accuracy.
`COCODetectionMetric`	Evaluation in COCO protocol.
`LossShow`	Show loss.
`MeanIOU`	Evaluation segmentation results.
`EvalMetric`	Base class for all evaluation metrics.
`VOC07MApMetric`	Mean average precision metric for PASCAL V0C 07 dataset.
`VOCMApMetric`	Calculate mean AP for object detection task.
`Kitti3DMetricDet`

6.1.5.5.2. API Reference¶

class hat.metrics.Accuracy(axis=1, name='accuracy')¶

Computes accuracy classification score.

参数

axis (int) – The axis that represents classes
name (str) – Name of this metric instance for display.

update(labels, preds)¶: Override this method to update the state variables.

class hat.metrics.AccuracySeg(name='accuracy', axis=1)¶

# TODO(min.du, 0.5): merged with Accuracy #.

update(output)¶: Override this method to update the state variables.

class hat.metrics.COCODetectionMetric(ann_file: str, val_interval: int = 1, name: str = 'COCOMeanAP', save_prefix: str = './WORKSPACE/results', adas_eval_task: Optional[str] = None, use_time: bool = True, cleanup: bool = False)¶

Evaluation in COCO protocol.

参数

ann_file – validation data annotation json file path.
val_interval – evaluation interval.
name – name of this metric instance for display.
save_prefix – path to save result.
adas_eval_task – task name for adas-eval, such as ‘vehicle’, ‘person’ and so on.
use_time – whether to use time for name.
cleanup – whether to clean up the saved results when the process ends.

引发

RuntimeError – fail to write json to disk.

get()¶: Get evaluation metrics.

reset()¶

Reset the metric state variables to their default value.

If （and only if） there are state variables that are not registered with self.add_state need to be regularly set to default values, please extend this method in subclasses.

update(output: Dict)¶

Update internal buffer with latest predictions.

Note that the statistics are not available until you call self.get() to return the metrics.

参数: output – A dict of model output which includes det results and image infos.

class hat.metrics.EvalMetric(name: Union[List[str], str], process_group: Optional[torch._C._distributed_c10d.ProcessGroup] = None, warn_without_compute: bool = True)¶

Base class for all evaluation metrics.

Built on top of torchmetrics.metric.Metric, this base class introduces the name attribute and a name-value format output (the get method). It also makes possible to syncnronize state tensors of different shapes in each device to support AP-like metrics.

注解

This is a base class that provides common metric interfaces. One should not use this class directly, but inherit it to create new metric classes instead.

参数

name – Name of this metric instance for display.
process_group – Specify the process group on which synchronization is called. Default: None (which selects the entire world)
warn_without_compute – Whether to output warning log if self.compute is not called in self.get. Since synchronization among devices is executed in self.compute, this value reflects if the metric will support distributed computation.

compute() → Union[float, List[float]]¶

Override this method to compute final results from metric states.

All states variables registered with self.add_state are synchronized across devices before the execution of this method.

get() → Tuple[Union[str, List[str]], Union[float, List[float]]]¶

Get current evaluation result.

To skip the synchronization among devices, please override this method and calculate results without calling self.compute().

返回: Name of the metrics. values: Value of the evaluations.
返回类型: names

get_name_value()¶

Return zipped name and value pairs.

返回: A (name, value) tuple list.
返回类型: List(tuples)

reset() → None¶

Reset the metric state variables to their default value.

If （and only if） there are state variables that are not registered with self.add_state need to be regularly set to default values, please extend this method in subclasses.

abstract update(*_: Any, **__: Any) → None¶: Override this method to update the state variables.

class hat.metrics.Kitti3DMetricDet(current_classes: List[str], compute_aos: bool = False, name: str = 'kitti3dAPDet', difficultys: Optional[List] = None)¶

get()¶

Get current evaluation result.

To skip the synchronization among devices, please override this method and calculate results without calling self.compute().

返回: Name of the metrics. values: Value of the evaluations.
返回类型: names

reset()¶

Reset the metric state variables to their default value.

If （and only if） there are state variables that are not registered with self.add_state need to be regularly set to default values, please extend this method in subclasses.

update(preds, labels)¶: Override this method to update the state variables.

class hat.metrics.LossShow(name: str = 'loss', norm: bool = True)¶

Show loss.

# TODO(min.du, 0.1): a better class name is required #

参数

name – Name of this metric instance for display.
norm – Whether norm loss when loss size bigger than 1. If True, calculate mean loss, else calculate loss sum. Default True.

get()¶

Get current evaluation result.

To skip the synchronization among devices, please override this method and calculate results without calling self.compute().

返回: Name of the metrics. values: Value of the evaluations.
返回类型: names

reset()¶

Reset the metric state variables to their default value.

If （and only if） there are state variables that are not registered with self.add_state need to be regularly set to default values, please extend this method in subclasses.

update(loss: Union[torch.Tensor, Dict[str, torch.Tensor]])¶: Override this method to update the state variables.

class hat.metrics.MeanIOU(seg_class: List[str], name: str = 'MeanIOU', ignore_index: int = 255, global_ignore_index: Union[Sequence, int] = 255, verbose: bool = False)¶

Evaluation segmentation results.

参数

seg_class (list(str)) – A list of classes the segmentation dataset includes，the order should be the same as the label.
name (str) – Name of this metric instance for display, also used as monitor params for Checkpoint.
ignore_index (int) – The label index that will be ignored in evaluation.
global_ignore_index (list,int) – The label index that will be ignored in global evaluation,such as:mIoU,mAcc,aAcc.Supporting list of label index.
verbose (bool) – Whether to return verbose value for aidi eval, default is False.

compute()¶: Get evaluation metrics.

update(label: torch.Tensor, preds: Union[Sequence[torch.Tensor], torch.Tensor])¶

Update internal buffer with latest predictions.

Note that the statistics are not available until you call self.get() to return the metrics.

参数

preds – model output.
label – gt.

class hat.metrics.TopKAccuracy(top_k, name='top_k_accuracy')¶

Computes top k predictions accuracy.

TopKAccuracy differs from Accuracy in that it considers the prediction to be True as long as the ground truth label is in the top K predicated labels.

If top_k = 1, then TopKAccuracy is identical to Accuracy.

参数

top_k (int) – Whether targets are in top k predictions.
name (str) – Name of this metric instance for display.

update(labels, preds)¶: Override this method to update the state variables.

class hat.metrics.VOC07MApMetric(num_classes: int, iou_thresh: float = 0.5, class_names: Optional[List[str]] = None)¶

Mean average precision metric for PASCAL V0C 07 dataset.

参数

num_classes – Num classs.
iou_thresh – IOU overlap threshold for TP
class_names – if provided, will print out AP for each class

class hat.metrics.VOCMApMetric(num_classes: int, iou_thresh: Union[float, List] = 0.5, class_names: Optional[List[str]] = None, ignore_ioa_thresh: float = 0.2, score_threshs: Optional[List[float]] = None, max_iou_thresh: Optional[float] = None, iou_thresh_interval: float = 0.05)¶

Calculate mean AP for object detection task.

参数

num_classes – Num classs.
iou_thresh – IOU overlap threshold for TP.
class_names – If provided, will print out AP for each class.
ignore_ioa_thresh – The IOA threshold for ignored GTs.
score_threshs – If provided, will print recall/precision at each score threshold.
max_iou_thresh – If provided, will calculate average AP at each iou threshold from ‘iou_thresh’ to ‘max_iou_thresh’, and the step is ‘iou_thresh_interval’. Must be larger than iou_thresh.
iou_thresh_interval – The step to generate a list of iou thresholds. Default is 0.05. You need to make sure max_iou_thresh - iou_thresh can be devided by this value.

compute()¶

Override this method to compute final results from metric states.

All states variables registered with self.add_state are synchronized across devices before the execution of this method.

gather_metrics()¶: Update num_inst and sum_metric.

reset()¶: Clear the internal statistics to initial state.

update(model_outs: Dict)¶

model_outs is a dict, the meaning of it’s key is as following.

pred_bboxes(List): Each element of pred_bboxes is the predict result: of an image. It’s shape is (N, 6), where 6 means (x1, y1, x2, y2, label, score).
gt_bboxes(List): Each element of gt_bboxes is the bboxes’ coordinates: of an image. It’s shape is (N, 4), where 4 means (x1, y1, x2, y2).
gt_classes(List): Each element of gt_classes is the bboxes’ classes: of an image. It’s shape is (N).
gt_difficult(List): Each element of gt_difficult is the bboxes’: difficult flag of an image. It’s shape is (N).