Model Evaluation
Model Evaluation is central to ensuring the quality and value of research results on the OnSite Learning Center. Standardized metrics and visualization tools help users understand model performance across datasets, including Precision, Recall, and F1 score. They also support parameter optimization, identification of bias or variance issues, and continuous model improvement.
Perception
For Perception, Model Evaluation metrics include mean average precision, Precision, and Recall. Note that not every task supports all of these metrics.
Calculate the results of the dataset data set after the perception taskMean Average Precision (mAP). This is one of the most commonly used metrics in object detection tasks, and it takes into account the precision and recall of the model. mAP is often calculated at multiple overlapping thresholds to evaluate the model's consistent performance in different situations.
Calculate the results of the dataset data set for perception problemsPrecision.Precision is measured as the ratio of positive examples correctly identified by the model to all positive examples identified as true positives divided by the sum of true positives and false positives.
Calculate the results of the dataset data set for perception problemsRecall。Recall rate refers to the ratio of positive cases correctly identified by the model to all actual positive cases, that is, true positives divided by the sum of true positives and false negatives.
Calculate the results of the dataset data set for perception problemsF1 Score。The F1 score is the harmonic mean of precision and recall, providing a single metric that balances the relationship between the two.
Calculate the results of the dataset data set for perception problemsOverlap Threshold.In target detection, an overlap threshold is set to determine whether the matching degree between the predicted box and the real box is sufficient. IoU (Intersection over Union) is usually used as a metric.
Calculate the results of the dataset data set for perception problemsMean Intersection over Union (mIoU).In semantic segmentation tasks, mIoU is used to evaluate the degree of overlap between predicted pixels and actual pixels, which is the average of the ratio of the intersection and union of the predicted area and the actual area.
Prediction
For Prediction, Model Evaluation metrics include ADE, FDE, MR, and others.
Calculate the results of the dataset data set for prediction tasksRoot Mean Square Error (RMSE): This is a metric widely used in regression tasks that measures the square root of the average squared difference between the predicted value and the true value. In trajectory prediction, it represents the square root of the average Euclidean distance between predicted trajectory points and actual trajectory points.
Calculate the results of the dataset data set for prediction tasksNegative Log-Likelihood (NLL).This metric is primarily used in the evaluation of probabilistic models, and it measures the difference between the probability distribution predicted by the model and the actual observations. In trajectory prediction, NLL focuses on evaluating the model's probability estimation accuracy for trajectory prediction, especially for models involving multi-modal prediction.
Calculate the results of the dataset data set for prediction tasksAverage Displacement Error (ADE).ADE calculates the average Euclidean distance between each predicted point and the actual point in the entire predicted trajectory. It provides an overall error measure indicating how much the predicted trajectory deviates from the true trajectory on average over the entire sequence.
Calculate the results of the dataset data set for prediction tasksFinal Displacement Error (FDE).FDE specifically focuses on the Euclidean distance between the last point of the predicted trajectory and the end point of the actual trajectory. It measures the accuracy of the model at the predicted end moment, which is especially important for scenarios that require accurate stopping points.
Calculate the results of the dataset data set for prediction tasksMiss Rate (MR).MR calculates the ratio of the Euclidean distance between the end point of the predicted trajectory and the end point of the actual trajectory that is greater than a given threshold (usually 2.0 meters). It reveals the overall accuracy of the model in predicting the end position of the trajectory.
Calculate the results of the dataset data set for prediction tasksOff-Road Rate (OffRoadRate): When the dataset contains map information, this metric counts the proportion of predicted trajectories that are not on a valid road network. It evaluates the model's performance in adhering to road rules.
Calculate the results of the dataset data set for prediction tasksMean average precision weighted by heading (mAPH): Derived from the Waymo data set, mAPH considers the predicted heading angle (direction) and position accuracy, and evaluates the overall performance of the model by calculating the area under the Precision-Recall curve. At the same time, the predicted heading angle is used as a weighting factor, which is suitable for tasks that require accurate direction prediction.