karlstoney.com

Creating an OpenWAF solution with Nginx, ElasticSearch and ModSecurity

Karl Stoney -

So many technologies in one title!

Recently I've been spending quite a bit of time investigating ModSecurity as a potential replacement Web Application Firewall, and I've had some really positive results. The purpose of this post is to share with you how I've set this up, so you can do something similar yourself. After all, who wouldn't want to be alerted to suspicious behaviour on their site in slack:

This post does presume you have a base level of understanding of Kubernetes, Docker and Fluentd.

Requirements

We have a WAF setup already, so the purpose of this piece of work was to see if we could get the same sort of capabilities with non-commercial (free) software, the sorts of things we were looking for were:

  • A comparable level of protection against the most common attacks
  • Results and associated metadata to be streamed into ElasticSearch for Engineers and Security Analysts to investigate
  • Alerting capability
  • Ability to make modifications as code, fitting into our CI/CD pipeline

Architecture

At a very high level, depicted in a very bad diagram, this is what we do:

The following specific components are being used:

  • We run Nginx on Kubernetes, and make use of the ingress-nginx-controller which comes pre-bundled with ModSecurity 3.0
  • We run fluentd as a sidecar container to the nginx-ingress-controller, and use this gem to parse the modsecurity files
  • We're running elasticsearch 6.2.0
  • For alerting into slack, we're using elastalert

So starting at the top...

Nginx & ModSecurity

As I mentioned before, running ingress-nginx-controller on top of kubernetes means enabling ModSecurity is simply a case of enabling the flag in your configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration-external
  namespace: ingress-nginx
data:
  enable-modsecurity: "true"
  enable-owasp-modsecurity-crs: "true"

You'll notice we have owasp-modsecurity-crs enabled as well, this will add the OWASP Core Ruleset rules into ModSecurity, which give you a pretty good base level of protection.

The next thing to do is to configure your ingress object to enable ModSecurity, and also to write the log files to a given location. We can do this with an annotation on the ingress, to inject a snippet into NGinx.

metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      modsecurity_rules '
        SecRuleEngine On
        SecAuditLog /var/log/modsec/audit.log
        SecAuditLogParts ABCIJDEFHZ
        SecAuditEngine RelevantOnly
        SecRuleRemoveById 932140
      ';

You can read more about these values in the ModSecurity Reference Manual, but in summary we're enabling "blocking mode" for bad request, and logging anything we block to /var/log/modsec/audit.log. We also disable rule 932140, because it was throwing too many false positives for us and isn't relevant to our application.

At this point in time, hitting your service with a malicious request (for example; sql injection) should return a 403:

❯ curl -s -o /dev/null -w '%{http_code}' "https://your-website.com?username=1'%20or%20'1'%20=%20'"
403
Fluentd

So now we're at a point where ModSecurity is effectively blocking requests it thinks are bad, and it will be writing the detail of those logs to /var/log/modsec/audit.log. That log isn't easily accessible (or readable) so we made the decision to use fluentd to parse, and send those logs to ElasticSearch where they can be visualised in Kibana, or alerted on by ElastAlert.

We run fluentd as a sidecar in the ingress-nginx pod. We share a volume mount between ingress-nginx and fluentd so that fluentd can access the modsecurity logs.

I've pushed up the code for our docker container here for those of you want to see it, and in your kubernetes deployment yaml for ingress-nginx you'll need to add a second container:

      - name: fluentd-modsec
        env:
        - name: ES_HOSTS
          value: your-elasticsearch-host
        - name: ES_USERNAME
          value: username
        - name: ES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: nginx-modsec-es
        image: eu.gcr.io/your-project/docker-modsec-fluentd:latest
        volumeMounts:
        - mountPath: /var/log/modsec
          name: modsec-logs
Elastalert

At this point, fluentd should be pushing your logs into a modsecurity index in elasticsearch:

Wondeful, the next thing to do is set up some alerting. For this, we're using another open source project called ElastAlert, by Yelp. I'm not going to go into setting up ElastAlert here, their documentation is pretty complete, but I will give you this sample alert which I what I used to create this blog post:

name: OWASP WAF Triggered
type: any

index: modsecurity-*
doc_type: doc

realert:
  minutes: 5

timeframe:
  minutes: 5

filter:
- term:
    violation.ver: "OWASP_CRS/3.0.0"

query_key:
 - "violation.msg"
 - "requestHeaders.host"

alert:
  - "slack"

use_kibana4_dashboard: "https://your-kibana.com/goto/524f4bae947ec5b2b879fc74d95f8869"
slack_emoji_override: ":hack:"
slack_msg_color: "danger"
slack_username_override: "ModSecurity"

alert_subject: "Potential malicious requests | <{0}|Dashboard>"
alert_subject_args:
  - kibana_link

alert_text: "Rule: '{0}'\nHost: '{1}'"
alert_text_type: alert_text_only
alert_text_args: ["violation.msg", "requestHeaders.host"]

And that's it! You'll have your Web Application Firewall logging to Elasticsearch, and alerting into Slack (or whatever medium you decide!