IP address filtering in Logstash: To cidr, or not to cidr?

This way (if conditions) of handling IP addresses works fine for a couple of static IP addresses. But you can see from the above condition: The more IP addresses need to be handled, the larger the condition becomes. That's not only annoying to manage, Logstash also needs more time to parse the condition (for each incoming event!).

Using if conditions with regular expressions

If all the application hosts are known and use static IP addresses, the if conditions can also be shortened by creating a regular expression (regex) around the IP addresses:

root@logstash:~# cat /etc/logstash/conf.d/10-filter.conf

filter {

  if [host] =~ /^192\.168\.253\.(111|112)/ or [host] =~ /^10.150.79.(139|140|143|144)/ {
    mutate { add_field => { "logtarget" => "test" } }
  } else if [host] =~ /^192\.168\.253\.(115|116)/ or [host] =~ /^10.150.79.1([1-9])/ {
    mutate { add_field => { "logtarget" => "prod" } }
  } else {
    mutate { add_field => { "logtarget" => "generic" } }
  }

}

This is basically the same filter as before, just shorter and (at least in my opinion) more manageable.

But there's still one problem with this filter: We must know the IP addresses of the application. If the application scales up, additional hosts may need to be added. The chances that new hosts are forgotten to be added into the filter are pretty high!

Using CIDR filter

There's one more filter plugin, which can help us out here: The CIDR filters plugin. The documentation of the plugin lacks a lot of information and examples and is at first hard to understand. Basically what you need to understand is the following:

• The "address" option is the most important input. The value of this option tells the filter plugin which IP address to compare against the "network" option. As the CIDR filter doesn't make sense without the "address" option, it should actually be marked as a mandatory option.
  
• The "network" option is an array that supports multiple networks in CIDR format (e.g. 192.168.10.0/24). The value from "address" is compared against the networks listed in "network".
  
• Instead of listing networks in the "network" option, an external file can also be used as "list of networks". The local path to this external file can be specified using the "network_path" option.
  
• If the "address" is a match against one of the CIDR networks listed in "network" (or "network_path"), additional actions inside the cidr {} filter are applied (such as add_field or remove_field).
  

"Translating" the above filters to use the cidr filter, results in the following new filter:

root@logstash:~# cat /etc/logstash/conf.d/10-filter.conf

filter {

  # Test networks
  cidr {
    address => [ "%{host}" ]
    network => [ "192.168.253.111/32", "192.168.253.112/32", "10.150.79.128/25" ]
    add_field => { "logtarget" => "test" }
  }


  # Production networks
  cidr {
    address => [ "%{host}" ]
    network => [ "192.168.253.115/32", "192.168.253.116/32", "10.150.79.0/25" ]
    add_field => { "logtarget" => "prod" }
  }

}

In this example, the [host] field is used as variable and used as value for the "address" option. "address" is then compared against different networks, defined in "network". Note that there are no if conditions in this situation; the comparison happens within the cidr filter and actions (add_field) are only applied if the comparison is a match (true).

Although each environment has two static host IP addresses defined (using the /32 suffix), the big advantage here is the separation of segmented networks. By checking the "address" against the TEST (10.150.79.128/25) and PROD (10.150.79.0/25) network ranges, we don't need to know the exact source IPs where the application runs. Application scaling (within the environment) is perfectly supported and the filter does not need to be adjusted.