Confusion matrix and how Defence cyber security use confusion matrix or its two types of error
Whenever we train any machine learning model or prediction model the main aim is to create a model which is the most accurate and gives the most accurate output.
after creating such models our first and only concern is that whether the model is predicting correctly? is it experienced enough?
Thus cross-validation is needed for any model,
to check these issue we split our data into a training set and testing set
later on, we use these to create a confusion matrix.
Now, What is the confusion Matrix?
The confusion matrix is simply a matrix that shows the parameters that how many value model is predicting or classifying correctly.
Our model can have two types of error
- Model classified positive but it’s not in real
- Model classified negative but it’s not in real
Which error is more misleading depends on the problem statement
If the model predicting positive and the actual class is positive too then it is a true positive.
If the model is predicting positive and the actual class is negative then it is a false negative. (Type 2)
If the model is predicting negative and the actual class is positive then it is a false positive. (Type 1)
If the model is predicting negative and the actual class is negative too then it is a true negative.
Let's take the case of Evaluating Machine Learning Classifiers for Defensive Cyberspace Operations,
Today’s defensive cyber sensors are dominated by signature-based analytical methods that require continuous maintenance and lack the ability to detect unknown threats. Anomaly detection offers the ability to detect unknown threats, DCO (defense cyberspace operations) relies on signature-based systems to detect threats against networks and computer systems. Signature-based systems perform well identifying known threats and achieve low false-positive rates when finely tuned signatures are used. However, these systems are incapable of detecting novel attacks for which no signatures exist.
Contributing challenges have been identified as (i) high cost of errors (time to investigate false alarms and cost of missed attacks); (ii) lack of labeled training data (ground truth); (iii) semantic gap between classification output and operational interpretation; (iv) variability in input data
These attack classifications are done using ML models,
Results from the empirical tests are reported as a matrix of correct and incorrect classifications, i.e. confusion matrix
where each classified data will be divided as True Positive, True Negative, false positive, and false negative
The main aim of training any data is accuracy, lowest FP rate, highest TP rate.
they derive various metrics based on the Confusion matrix such as
Accuracy = TP + TN / TP + TN + FP + FN
Misclassification = FP + FN / TP + TN + FP + FN
Precision = TP / TP + FP
Sensitivity / Recall = TP / TP + FN
Specificity = TN / TN + FP
Based on these they make further decisions on sensitivity and accuracy of threats detection models.
Hope you all find it informative and interesting..!!!