Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

test: add containerized default go-ftw tests to docker compose file#4627

Open
studersi wants to merge 2 commits intocoreruleset:maincoreruleset/coreruleset:mainfrom
studersi:feature/default-testing-setupstudersi/coreruleset:feature/default-testing-setupCopy head branch name to clipboard
Open

test: add containerized default go-ftw tests to docker compose file#4627
studersi wants to merge 2 commits intocoreruleset:maincoreruleset/coreruleset:mainfrom
studersi:feature/default-testing-setupstudersi/coreruleset:feature/default-testing-setupCopy head branch name to clipboard

Conversation

@studersi
Copy link
Copy Markdown
Contributor

@studersi studersi commented May 1, 2026

Proposed changes

This pull request adds go-ftw containers to the docker-compose.yml file with default configurations for getting started more quickly with ruleset testing.

PR Checklist

  • I have read the CONTRIBUTING doc
  • (does not apply) I have added positive tests proving my fix/feature works as intended.
  • (does not apply) I have added negative tests that prove my fix/feature considers common cases that might end in false positives
  • (does not apply) In case you changed a regular expression, you are not adding a ReDOS for pcre. You can check this using regexploit
  • (does not apply) My test use the comment field to write the expected behavior
  • I have added documentation for the rule or change (when appropriate)

Further comments

I find it useful sometimes to be able to quickly spin up a completely ephemeral testing setup by just cloning the repo and running docker-compose up, without having to install anything and preparing specific configuration files etc.

If this pull request is at odds with other workflows that are used by the community, feel free to close it.

AI Disclosure

AI usage for this contribution: Used

If "Used", complete the details below:

Detail Response
Tool(s) used Jetbrains Junie with GTP 5.3 codex
What was AI-assisted initial scaffolding for test setup
Review performed yes

For the reviewer

  • Positive and negative tests were added
  • Tests cover the intended fix/feature properly
  • No usage of dangerous constructs like ctl:requestBodyAccess=Off were used in the rule
  • In case a regular expression was changed, there is no ReDOS
  • Documentation is clear for the rule/change
  • If a contribution shows signs of unreviewed AI generation (e.g., plausible-but-broken regex, generic boilerplate comments, hallucinated SecLang directives), reviewers should ask about AI usage regardless of what was checked.

@franbuehler
Copy link
Copy Markdown
Contributor

franbuehler commented May 2, 2026

Thank you very much for this contribution!

I tested it locally and it works for me:

Apache:

$ docker-compose -f tests/docker-compose.yml run --rm ftw-modsec2-apache
[+] Creating 2/2
 ✔ Container tests-backend-1  Running                                                                            0.0s 
 ✔ Container modsec2-apache   Running                                                                            0.0s 
[+] Running 4/4
 ✔ ftw-modsec2-apache Pulled                                                                                     1.8s 
   ✔ d8ad8cd72600 Pull complete                                                                                  0.8s 
   ✔ 04b1a4f1f998 Pull complete                                                                                  0.8s 
   ✔ ae8258558227 Pull complete                                                                                  0.9s 
🛠️ Starting tests!
🚀 Running go-ftw!

➕ run 4644 total tests in 35.944717046s
⏭ skipped 0 tests
☝ ignored 2 tests
🎉 All tests successful!

I'm wondering why these tests are failing with NGINX:

$ docker-compose -f tests/docker-compose.yml run --rm ftw-modsec3-nginx
[+] Creating 2/2
 ✔ Container tests-backend-1  Running                                                                            0.0s 
 ✔ Container modsec3-nginx    Running                                                                            0.0s 
🛠️ Starting tests!
🚀 Running go-ftw!

💥 951170-1 failed in 2.827913ms (RTT 51.460598ms)
💥 951240-1 failed in 3.63687ms (RTT 52.113696ms)
💥 951240-2 failed in 3.181335ms (RTT 51.636204ms)
💥 951260-1 failed in 3.250619ms (RTT 51.827101ms)
💥 954120-1 failed in 2.99727ms (RTT 51.560796ms)
💥 954120-2 failed in 3.011686ms (RTT 51.629663ms)
3:17PM INF Retrying test once: 959100-1
Error: retry-once
exit status 1

@studersi
Copy link
Copy Markdown
Contributor Author

studersi commented May 2, 2026

Thanks for having a look at this. In my case, the only failing tests I had were the ones with exceptions in tests/regression/httpd-overrides.yaml and tests/regression/nginx-overrides.yaml. I added these exceptions to my corresponding tests/regression/ftw-modsec2-apache.yaml and tests/regression/ftw-modsec3-nginx.yaml config files. Aside from those I did not have any failed tests.

I just rechecked with the latest version of the main branch and there are no failed tests in my case:

coreruleset$ docker compose -f tests/docker-compose.yml up -d modsec2-apache; \
> docker compose -f tests/docker-compose.yml run --rm ftw-modsec2-apache; \
> docker compose -f tests/docker-compose.yml down
[+] up 3/3
 ✔ Network tests_default     Created                                                                                                                                                                                                                             0.1s
 ✔ Container tests-backend-1 Started                                                                                                                                                                                                                             0.5s
 ✔ Container modsec2-apache  Started                                                                                                                                                                                                                             0.6s
[+]  2/2t 2/22
 ✔ Container tests-backend-1 Running                                                                                                                                                                                                                             0.0s
 ✔ Container modsec2-apache  Running                                                                                                                                                                                                                             0.0s
Container tests-ftw-modsec2-apache-run-d8d73824f1d7 Creating 
Container tests-ftw-modsec2-apache-run-d8d73824f1d7 Created 
🛠️ Starting tests!
🚀 Running go-ftw!
➕ run 4738 total tests in 57.345812646s
⏭ skipped 0 tests
☝ ignored 2 tests
🎉 All tests successful!
[+] down 3/3
 ✔ Container modsec2-apache  Removed                                                                                                                                                                                                                            10.3s
 ✔ Container tests-backend-1 Removed                                                                                                                                                                                                                            0.2s
 ✔ Network tests_default     Removed

coreruleset$ docker compose -f tests/docker-compose.yml up -d modsec3-nginx; \
> docker compose -f tests/docker-compose.yml run --rm ftw-modsec3-nginx; \
> docker compose -f tests/docker-compose.yml down
[+] up 2/2
 ✔ Container tests-backend-1 Running                                                                                                                                                                                                                             0.0s
 ✔ Container modsec3-nginx   Started                                                                                                                                                                                                                             0.3s
[+]  2/2t 2/22
 ✔ Container tests-backend-1 Running                                                                                                                                                                                                                             0.0s
 ✔ Container modsec3-nginx   Running                                                                                                                                                                                                                             0.0s
Container tests-ftw-modsec3-nginx-run-a64b77fe6d4a Creating 
Container tests-ftw-modsec3-nginx-run-a64b77fe6d4a Created 
🛠️ Starting tests!
🚀 Running go-ftw!
➕ run 4738 total tests in 37.776262689s
⏭ skipped 0 tests
☝ ignored 88 tests
🎉 All tests successful!
[+] down 3/3
 ✔ Container modsec3-nginx   Removed                                                                                                                                                                                                                            10.3s
 ✔ Container tests-backend-1 Removed                                                                                                                                                                                                                            0.2s
 ✔ Network tests_default     Removed

Could it be that you are not running this on the latest version of the main branch or do you have customized tests because of ongoing development?

@franbuehler
Copy link
Copy Markdown
Contributor

franbuehler commented May 3, 2026

This is strange. I checked out your feature branch default-testing-setup. So I should get the same results as you.
Maybe someone else could have a look.
But besides of that, it looks good for me!

@studersi
Copy link
Copy Markdown
Contributor Author

studersi commented May 3, 2026

I rebased the feature branch onto the latest state of main, could you try again? in my case, it still shows no errors.

@franbuehler
Copy link
Copy Markdown
Contributor

franbuehler commented May 3, 2026

Thank you for rebasing.
This is still very strange. The tests with the Apache containers always succeed.
But the tests with the NGINX containers are failing. I've noticed that it's not always the same tests that fail. Different tests fail in each run and the number of failed tests also varies.
Does anyone else has the same problem?

@theseion theseion mentioned this pull request May 4, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@studersi @franbuehler
Morty Proxy This is a proxified and sanitized view of the page, visit original site.