Similiar to the famous saying “Don’t drink and drive”, you could also say “Don’t mix fail2ban and OpenVAS”. What’s the point?

fail2ban is an open source intrusion detection system, which takes action (typically, changing the firewall) in real-time in case that certain string patterns are found in one or several log files. A very common use case for it is when a (non-distributed) Denial of Service attack is performed on your webserver, the requests are being traced in the access_log file of your webserver process. Based on a threshold of maximal number of requests allowed within a given timeframe, the IP, from which the requests are coming, may be blocked via a firewall rule. Typically, these rules have some sort of expiration mechanism and thus will be purged automatically after a short period of time (typically several minutes).

OpenVAS is an open source vulnerability scanner and manager. Based on a positive list of known vulnerabilities, which are known to be available in the wild, checks are performed on a target system. Based on the severity of the findings, the result is being aggregated in a dedicated report. OpenVAS today contains out of more than 35,000 Network Vulnerability Tests (NVTs), especially covering those security related publications made by CVE and DFN-CERT — amongst them the well-known OpenSSL heartbleed attack.

To understand why mixing those two tools might not be such a good idea, we need to look at bit closer on how the OpenVAS works structurally: At first glance, a typical scan by OpenVAS executes the linux program nmap on the target. Nmap is a portscanner tool which tries to detect which ports are open and subject to a vulernability scan. On the list of open ports, the OpenVAS Scanner then performs appropriate checks via the already mentioned NVTs. As there are usually multiple possible security flaws for one protocol/application, multiple checks need to be performed in a sequence within a short timeframe. Those checks all typically also do not comply to a usual pattern of requests which makes them suspicious to any kind of intrusion detection system. Furthermore, the high frequency of requests makes OpenVAS look very similar to an attacker trying to provoke a denial of service situation. Thus, fail2ban easily may mistake the scanning by OpenVAS as an attacker and trigger corresponding countermeasures, especially including those to ban the IP temporarily.

During NVT 3 the fail2ban block is activated and thus jeopardizes the result of this NVT and all subsequent NVTs

This, on the other hand, jeopardizes the result of the scan, as the target system appears to be no longer reachable by the OpenVAS server. The unfortunate part of the OpenVAS is that it does not detect this situation, but simply states that “no vulnerability” was found for those NVTs which were no longer able to reach the target system. From the perspective of an end user this may look very strangely: Executing the same set of tests multiple times – even without changing anything on the target system – may lead to different reports by OpenVAS. This may cast doubt on the reliability of the scanning tool.

Depending on the time fail2ban has triggered the blocking of the OpenVAS' IP, different NVTs may be affected, which explains the differences in two subsequent scans

Based on the charts above, you may conclude some further aspects of this situation:

1. Depending on when fail2ban performs the ban, the NVT currently running at this point in time may produce a inappropriate result.
2. There may be tests which never will may reach the target system (in the example above: NVT 4). This means, that they will also never report any vulnerability of the system, even if the system is subject to this attack vector.

Especially the latter conclusion is very alarming. In plain words this means that if the target system is vulnerable to what the NVT 4 is checking, OpenVAS will never detect it, but an attacker simply trying it would be able to exploit the security gap. In short, fail2ban may make OpenVAS blind for an unpredictable set of tests.

So, what’s the cure for it? Fortunately, there is simple one for it: Disable fail2ban for your OpenVAS’ system! This is called whitelisting and can be done with the configuration option ignoreip.