Adding Automated Penetration Testing to CI Pipelines

CI Pipelines: adding penetration testing
Testing, particularly around security, is a core part of the ethos of all NearForm development teams.

In many organisations, penetration testing can often happen just before a product first pushes to production, and periodically thereafter.

Penetration testing is performed by external teams and is focused on finding vulnerabilities that may exist within the infrastructure or code base. It can be a slow and expensive process.

This brings up two questions:

  1. Can we automate some basic penetration testing?
  2. Can we continuously security test as we develop?

Owasp Zed Attack Proxy

Open Web Application Security Project – OWASP is the gold standard of tools, advice and security best practices.

We will focus on using ZED Attack Proxy – ZAP – and show how to integrate it into our Continuous Integration (CI) pipeline. The goal is to automate ZAP with as little configuration as possible.

Setting up CI

Jenkins users (and Jira ones also) may benefit from the ready-to-use Jenkins plugin for your CI needs.

If the Jenkins plugin is not an option, ZAP has Docker support and a wiki full of interesting and useful information and instructions.

Note: The ZAP project also has a desktop multi-platform tool that can help test and prepare for the CI integration (or manual analyses) and also helps to get a feeling for the tools capabilities and configuration options.

Although ZAP excels at crawling and scanning frontend (SPA) applications, in this example we integrate ZAP with Udaru, an open source access manager for Node.js written by NearForm.

Udaru uses Swagger to document the endpoints and this is very useful when testing it with ZAP.

ZAP has two scripts which can help us accomplish this goal.

Note: ZAP docker images are quite big and download can be slow. You may notice there is a smaller and more CI optimized image called owasp/zap2docker-bare, however avoid it for now as it does not have necessary scripts to perform the scans needed.

The following scripts help to perform a Baseline Scan aand an API scan.

First, pull the weekly docker image

docker pull owasp/zap2docker-weekly

Run the baseline scan against a local http server, for example running on 8000 port

docker run -v $(pwd):/zap/wrk/:rw \
	owasp/zap2docker-weekly \
	-c baseline-scan.conf \
	-t https://$(ifconfig en0 | grep "inet " | cut -d " " -f2):8080
            -r baseline-scan-report.html

Run the API scan against the local server running on 8080 port based on the Swagger definition of the API.

docker run -v $(pwd):/zap/wrk/:rw \
	-t owasp/zap2docker-weekly \
	-c api-scan.conf \
	-t  https://$(ifconfig en0 | grep "inet " | cut -d " " -f2):8080/swagger.json \
	-f openapi \
            -r api-scan-report.html

Note: The command $(ifconfig en0 | grep "inet " | cut -d " " -f2) is used to get the correct IP address to be able to access the Site/API outside the docker container.

If you are on a MAC OSX this may not work as Docker has some difference in networking. Then you can use docker.for.mac.localhost for the host part of the -t argument.

Neither the baseline-scan.conf or the api-scan.conf files exist – the first time you run the commands use -ginstead of -c, which will produce the configuration files with levels of alerts set to warn.

docker run -v $(pwd):/zap/wrk/:rw \
	owasp/zap2docker-weekly \
	-g baseline-scan.conf \
	-t https://$(ifconfig en0 | grep "inet " | cut -d " " -f2):8080
docker run -v $(pwd):/zap/wrk/:rw \
	-t owasp/zap2docker-weekly \
	-g api-scan.conf \
	-t  https://$(ifconfig en0 | grep "inet " | cut -d " " -f2):8080/swagger.json \
	-f openapi

With these files generated, you can set which tests should fail/ignore/warn as you run the scripts.

The first few lines of the file describe the contents and their configuration.

Example from zap-baseline configuration file
10010	WARN	(Cookie No HttpOnly Flag)
10011	WARN	(Cookie Without Secure Flag)
10012	WARN	(Password Autocomplete in Browser)
10015	WARN	(Incomplete or No Cache-control and Pragma HTTP Header Set)
10016	WARN	(Web Browser XSS Protection Not Enabled)

Example from zap-api-scan configuration file
40014	WARN	(Cross Site Scripting (Persistent) - Active/release)
40016	WARN	(Cross Site Scripting (Persistent) - Prime - Active/release)
40017	WARN	(Cross Site Scripting (Persistent) - Spider - Active/release)
40018	WARN	(SQL Injection - Active/release)
40019	WARN	(SQL Injection - MySQL - Active/beta)
40020	WARN	(SQL Injection - Hypersonic SQL - Active/beta)
40021	WARN	(SQL Injection - Oracle - Active/beta)
40022	WARN	(SQL Injection - PostgreSQL - Active/beta)
40023	WARN	(Possible Username Enumeration - Active/beta)

Both of these scripts will test a front-end or back-end application. However, problems can arise with authenticating a back-end API request as this is a common case for testing REST APIs; this is usually the Authorization header.

This part is described in the ZAP blog and basically boils down to adding some extra configuration for the ZAP’s replacer add-on.

Example using ZAP API scan with Authorization header:

docker run -v $(pwd):/zap/wrk/:rw \
	-t owasp/zap2docker-weekly \
	-c api-scan.conf \
	-t  https://$(ifconfig en0 | grep "inet " | cut -d " " -f2):8080/swagger.json \
	-f openapi \
            -r api-scan-report.html
            -z "-config replacer.full_list\(0\).description=auth1 \
                 -config replacer.full_list\(0\).enabled=true \
                 -config replacer.full_list\(0\).matchtype=REQ_HEADER \
                 -config replacer.full_list\(0\).matchstr=Authorization \
                 -config replacer.full_list\(0\).regex=false \
                 -config replacer.full_list\(0\).replacement="

This technique can be used to add additional headers if needed.

The beauty of being able to pass a Swagger json to ZAP is that it significantly helps with things like identifying all endpoints to be tested, saves from building elaborate contexts for testing and makes it extremely easy to use thanks to the OpenAPI standardization (Swagger).

If there is no similar setup on the project that needs testing then this considerably adds to the time needed to do the configuration of a context needed to execute the scans and get some meaningful results.

Usefulness in the CI Pipeline?

If we think of ZAP pen-testing as a step in the CI execution pipeline (alongside checks, test runs and deploys) then baseline scanning fits here perfectly.

It is fast (length controllable with -d--duration options and defaults to 1 minute) safe and can provide basic insight during regular CI execution.

API scan (also referred to as attack in the documentation), is very comprehensive (with out of the box generated configuration), but it is CPU intensive and, depending on your API size (number of endpoints to test), very lengthy.

Also you need to factor in the fact that a big Docker image for ZAP needs to be downloaded (it is recommended to use the weekly one).

This does not really make it a good candidate for a step in the regular CI pipeline, but instead it is more suited to a dedicated CI job that runs nightly against a development or staging environment.

The scan is not considered safe so it would be prudent to run it on environments that can afford this kind of pressure (usually pre-production ones).

Reports generated from ZAP usually need manual overview until the issues detected are fixed.

It is prudent to do a manual execution of the scans and fix the initial issues prior to putting in any CI system.

Using ZAP with End to End Tests

ZAP can serve as man-in-the-middle .

This is very useful in the context of end-to-end tests or even integration tests that a project might already have.

Those tests might expose some vulnerabilities that cannot be crawled or found otherwise with baseline ZAP scan.

To make this work, we need to make sure ZAP is running as a proxy server, with network traffic from the tests runs through it and using a headless mode that CI systems can work with.

ZAP can be run via docker in headless mode on localhost like this:

docker run -d -v $(pwd):/zap/wrk/:rw -u zap -p 8080:8080 -i owasp/zap2docker-weekly -daemon -host -port 8080 -config* -config api.addrs.addr.regex=true

In tests that have the capability to use a proxy server, set the proxy option and let the test run as normal.

Example Using Karma.js

// karma.conf.js
module.exports = function(config) {
    browsers: ['ZAPChrome'],

    // you can define custom flags
    customLaunchers: {
      ZAPChrome: {
        base: 'ChromeHeadless',
        flags: ['--proxy-server=']
       // do not add --headless to Chrome as it will not allow karma to start running tests
### Example using Nightmare.js

// remember to npm i nightmare before running

const Nightmare = require('nightmare')
const nightmare = Nightmare({
  switches: {
    'proxy-server': '',
    'ignore-certificate-errors': true
  show: false // no UI action in CI env

  .type('#search_form_input_homepage', 'github nightmare')
  .wait('#r1-0 a.result__a')
  .evaluate(() => document.querySelector('#r1-0 a.result__a').href)
  .catch(error => {
    console.error('Search failed:', error)

ZAP will analyse the requests in its separate threads without affecting the tests and communication with them.

So far so good; we can put all of this in the CI pipeline since it is all headless and automated.

Getting Actionable Results

The final phase is to actually get some data or alerts or even reports from the ZAP based on the test run.

This is the hard part, there is nothing that works out of the box and our data is stuck in the ZAP docker container!

There are two approaches here to access the report data:

First, use the command line options and use the session and last_report flags to create a usable report, with one extra docker run like:

docker run -v $(pwd):/zap/wrk/:rw -u zap -p 8080:8080 -i owasp/zap2docker-weekly  -host -port 8080 -config* -config api.addrs.addr.regex=true -last_scan_report /zap/wrk/report.html -session /zap/wrk/newsession -cmd

Unfortunately, this means that it is necessary to preserve session information outside of the container and reuse it.

This is not very CI friendly, but doable if the end goal is the report.

Session data should also be preserved across CI steps and subsequent runs which needs manual setup and testing on most modern cloud CI providers.

A second approach is to use the ZAP API to get some data outside of ZAP’s database, which means the container needs to be running until data has been extracted.

We did not fully try these options out, but to help get some actionable result, we could use a Node.js client on NPM called zaproxy.

It’s a generated API client so some additional development work would be needed to get through to the report data.

There seems to be one more NPM option and it’s a Grunt runner called grunt-zaproxy.

This looks a bit more promising based on the example in the NPM readme. It should allow you to run scans and get a report, via Grunt. However it requires a bit of an update as it does not support latest Node.js versions.

It’s still worth noting that setting ZAP as proxy for tests is a valid approach, just maybe not for CI.

After the test runs, the results could be gathered and reviewed manually using the ZAP desktop application.

Integrating ZAP with Udaru

With all these known pros and cons how did we integrate ZAP in our CI pipeline on Udaru?

The short answer is that we did not. Not in the fully automated CI way.

We did create some npm scripts to run the baseline and API scans and we store the reports within our project documentation.

Anyone working on the project is free to run the scans to see if their work has disclosed a security vulnerability.

The flexibility of npm scripts allows us to easily integrate this into a CI pipeline, should we choose to perform a round of penetration testing (for example as a pre-release step).

The npm scripts basically start the Udaru server on a specified port, run the docker commands to initiate the scans and store the reports.

For more in depth information on the Udaru implementation, please see this pull request.


OWASP ZAP is a powerful tool in the battlefield of secure web applications.

The toolset developed around it is powerful, modern and is the cornerstone of moving to a fully automated penetration testing state in the CI Pipeline.

The Jenkins plugin is highly recommended for baseline scans.

For API scans, be prepared to invest a bit more into tooling environments, issue/alert analysis and fixes as this is something that is hard to fully automate.

Having peace of mind around the security of the product/project is definitely worth the investment of having this nice tool in your security belt.

For more information about our open source project Udaru see our article on dynamic intrusion detection for authorisation systems like Udaru.

Image: Jonatan Pie

Share Me

Related Reading


Don’t miss a beat

Get all the latest NearForm news, from technology to design. Sign up for our newsletter.

Follow us for more information on this and other topics.