Detecting Malicious DNS over HTTPS (DoH) Connections via Machine Learning Techniques

Abstract

DoH is a modern protocol used as an alternative to the existing DNS protocol, which provides confidentiality and integrity to DNS functions by using protected channels. Since this kind of connection can passthrough the current protection systems, it can be used for spreading malicious software. There is a need to find defense mechanisms that can detect and prevent these forms of malicious behaviors. In this study, we will concentrate on detecting malicious activities that are performed on the new DoH protocol using supervised machine learning techniques. Our study involves twelve different machine learning classifiers. The results were promising in employing machine learning techniques in this job since the accuracy scores were about 1 in detecting malicious DoH connections. In order to provide optimum computational and time costs in the proposed model, we used eight different techniques to calculate features importance and proposed a way to choose the meaningful features for our proposed detection model. This proposed way in selecting features helped in reaching the same accuracy if not improving the accuracy when the whole features are used.