Version

CodeQL CLI 2.5.0

Description of the issue

Possibly related to #5594

When using codeql test run and an options file containing semmle-extractor-options with -source or -target with an invalid value (e.g. -source 2; apparently javac expects -source 1.2) the extractor succeeds despite javac reporting errors.

Note that this does not apply to other incorrect argument usage, such as specifying both --source and --release or using the correct version notation, e.g. -source 1.2. In that case the extractor fails, as expected.

Reproduction steps

1. Create a query test file and a Java source file (content does not matter)
2. Create an options file containing:

   //semmle-extractor-options: --javac-args -source 2
   

3. Run codeql test run --keep-databases
   ❌ The test succeeds (or fails on RESULT stage)