evaluation methodology

Machine learning intrusion detection is evaluated almost entirely through metrics computed across an entire test partition. This paper argues that the practice has two defects and demonstrates both with measurements. The first defect is protocol-level concealment: network protocols carry attack traf ic in very dif erent proportions, so an aggregate metric averages over problems of unequal base rate, and a detector that fails on a minority protocol can still report a respectable aggregate figure. The second defect is prevalence distortion: precision depends on the base rate of the positive class, and benchmark corpora carry attack prevalences one to two orders of magnitude above those of operational networks, so reported precision overstates what a deployed system will achieve. Six experiments were conducted, crossing three detectors with two benchmark corpora and evaluating 72,856 held-out records. A Random Forest reporting an aggregate recall of 0.7626 was found to attain a recall of 0.0361 within the ICMP subset, a dif erence of a factor of 21.1 that no aggregate statistic exposes. Detection quality across the five assessable protocol subsets correlated with subset attack prevalence at r = 0.7773. Holding measured sensitivity and specificity fixed, a precision of 0.9793 at benchmark prevalence corresponds to 0.3415 at an operational prevalence of one percent. The paper closes with a reporting protocol that addresses both defects and costs no additional data, training, or computation.