Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 9e643c0

Browse filesBrowse files
lucasssvazlucasssvaz
authored
fix(testing): Checkout proper branch for Wokwi tests and small QoL improvements (#10435)
* fix(tests): Add missing newlines to output * fix(tests): Improve fibonacci test * fix(tests): Remove redundant targets from json * fix(wokwi): Checkout proper base branch for wokwi tests * feat(logging): Add logging to some tests to improve debugging * fix(ci): Make CI more permissive and improve messages * fix(tests): Bump pytest-embedded version to support P4
1 parent a7d9b67 commit 9e643c0
Copy full SHA for 9e643c0

File tree

Expand file treeCollapse file tree

18 files changed

+143
-64
lines changed
Filter options
Expand file treeCollapse file tree

18 files changed

+143
-64
lines changed

‎.github/scripts/tests_matrix.sh

Copy file name to clipboard
+26Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#!/bin/bash
2+
3+
build_types="'validation'"
4+
hw_types="'validation'"
5+
wokwi_types="'validation'"
6+
qemu_types="'validation'"
7+
8+
if [[ $IS_PR != 'true' ]] || [[ $PERFORMANCE_ENABLED == 'true' ]]; then
9+
build_types+=",'performance'"
10+
hw_types+=",'performance'"
11+
#wokwi_types+=",'performance'"
12+
#qemu_types+=",'performance'"
13+
fi
14+
15+
targets="'esp32','esp32s2','esp32s3','esp32c3','esp32c6','esp32h2'"
16+
17+
mkdir -p info
18+
19+
echo "[$wokwi_types]" > info/wokwi_types.txt
20+
echo "[$targets]" > info/targets.txt
21+
22+
echo "build-types=[$build_types]" >> $GITHUB_OUTPUT
23+
echo "hw-types=[$hw_types]" >> $GITHUB_OUTPUT
24+
echo "wokwi-types=[$wokwi_types]" >> $GITHUB_OUTPUT
25+
echo "qemu-types=[$qemu_types]" >> $GITHUB_OUTPUT
26+
echo "targets=[$targets]" >> $GITHUB_OUTPUT

‎.github/workflows/dangerjs.yml

Copy file name to clipboardExpand all lines: .github/workflows/dangerjs.yml
+3-1Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,6 @@ jobs:
1919
- name: DangerJS pull request linter
2020
uses: espressif/shared-github-dangerjs@v1
2121
env:
22-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
22+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
23+
rule-max-commits: 'false'
24+
commit-messages-min-summary-length: '10'

‎.github/workflows/tests.yml

Copy file name to clipboardExpand all lines: .github/workflows/tests.yml
+17-21Lines changed: 17 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -57,29 +57,25 @@ jobs:
5757
hw-types: ${{ steps.set-matrix.outputs.hw-types }}
5858
wokwi-types: ${{ steps.set-matrix.outputs.wokwi-types }}
5959
qemu-types: ${{ steps.set-matrix.outputs.qemu-types }}
60+
targets: ${{ steps.set-matrix.outputs.targets }}
61+
env:
62+
IS_PR: ${{ github.event.pull_request.number != null }}
63+
PERFORMANCE_ENABLED: ${{ contains(github.event.pull_request.labels.*.name, 'perf_test') }}
6064
steps:
65+
- name: Checkout
66+
uses: actions/checkout@v4
67+
with:
68+
sparse-checkout: .github/scripts/tests_matrix.sh
69+
6170
- name: Set matrix
6271
id: set-matrix
63-
run: |
64-
build_types='["validation"'
65-
hw_types='["validation"'
66-
wokwi_types='["validation"'
67-
qemu_types='["validation"'
68-
69-
is_pr=${{ github.event.pull_request.number != null }}
70-
is_performance_enabled=${{ contains(github.event.pull_request.labels.*.name, 'perf_test') }}
71-
72-
if [[ $is_pr != 'true' ]] || [[ $is_performance_enabled == 'true' ]]; then
73-
build_types+=',"performance"'
74-
hw_types+=',"performance"'
75-
#wokwi_types+=',"performance"'
76-
#qemu_types+=',"performance"'
77-
fi
72+
run: bash .github/scripts/tests_matrix.sh
7873

79-
echo "build-types=$build_types]" >> $GITHUB_OUTPUT
80-
echo "hw-types=$hw_types]" >> $GITHUB_OUTPUT
81-
echo "wokwi-types=$wokwi_types]" >> $GITHUB_OUTPUT
82-
echo "qemu-types=$qemu_types]" >> $GITHUB_OUTPUT
74+
- name: Upload
75+
uses: actions/upload-artifact@v4
76+
with:
77+
name: matrix_info
78+
path: info/*
8379

8480
call-build-tests:
8581
name: Build
@@ -88,7 +84,7 @@ jobs:
8884
strategy:
8985
matrix:
9086
type: ${{ fromJson(needs.gen-matrix.outputs.build-types) }}
91-
chip: ['esp32', 'esp32s2', 'esp32s3', 'esp32c3', 'esp32c6', 'esp32h2']
87+
chip: ${{ fromJson(needs.gen-matrix.outputs.targets) }}
9288
with:
9389
type: ${{ matrix.type }}
9490
chip: ${{ matrix.chip }}
@@ -105,7 +101,7 @@ jobs:
105101
fail-fast: false
106102
matrix:
107103
type: ${{ fromJson(needs.gen-matrix.outputs.hw-types) }}
108-
chip: ['esp32', 'esp32s2', 'esp32s3', 'esp32c3', 'esp32c6', 'esp32h2']
104+
chip: ${{ fromJson(needs.gen-matrix.outputs.targets) }}
109105
with:
110106
type: ${{ matrix.type }}
111107
chip: ${{ matrix.chip }}

‎.github/workflows/tests_results.yml

Copy file name to clipboardExpand all lines: .github/workflows/tests_results.yml
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ jobs:
7979
event_name: ${{ env.original_event }}
8080
files: ./artifacts/**/*.xml
8181
action_fail: true
82+
compare_to_earlier_commit: false
8283

8384
- name: Fail if tests failed
8485
if: ${{ env.original_conclusion == 'failure' || env.original_conclusion == 'timed_out' || github.event.workflow_run.conclusion == 'failure' || github.event.workflow_run.conclusion == 'timed_out' }}

‎.github/workflows/tests_wokwi.yml

Copy file name to clipboardExpand all lines: .github/workflows/tests_wokwi.yml
+32-4Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ jobs:
2222
outputs:
2323
pr_num: ${{ steps.set-ref.outputs.pr_num }}
2424
ref: ${{ steps.set-ref.outputs.ref }}
25+
base: ${{ steps.set-ref.outputs.base }}
26+
targets: ${{ steps.set-ref.outputs.targets }}
27+
types: ${{ steps.set-ref.outputs.types }}
2528
steps:
2629
- name: Report pending
2730
uses: actions/github-script@v7
@@ -51,10 +54,18 @@ jobs:
5154
name: event_file
5255
path: artifacts/event_file
5356

57+
- name: Download and extract matrix info
58+
uses: actions/download-artifact@v4
59+
with:
60+
github-token: ${{ secrets.GITHUB_TOKEN }}
61+
run-id: ${{ github.event.workflow_run.id }}
62+
name: matrix_info
63+
path: artifacts/matrix_info
64+
5465
- name: Try to read PR number
5566
id: set-ref
5667
run: |
57-
pr_num=$(jq -r '.pull_request.number' artifacts/event_file/event.json)
68+
pr_num=$(jq -r '.pull_request.number' artifacts/event_file/event.json | tr -cd "[:digit:]")
5869
if [ -z "$pr_num" ] || [ "$pr_num" == "null" ]; then
5970
pr_num=""
6071
fi
@@ -64,11 +75,22 @@ jobs:
6475
ref=${{ github.ref }}
6576
fi
6677
67-
action=$(jq -r '.action' artifacts/event_file/event.json)
78+
action=$(jq -r '.action' artifacts/event_file/event.json | tr -cd "[:alpha:]_")
6879
if [ "$action" == "null" ]; then
6980
action=""
7081
fi
7182
83+
base=$(jq -r '.pull_request.base.ref' artifacts/event_file/event.json | tr -cd "[:alnum:]/_.-")
84+
if [ -z "$base" ] || [ "$base" == "null" ]; then
85+
base=${{ github.ref }}
86+
fi
87+
88+
types=$(cat artifacts/matrix_info/wokwi_types.txt | tr -cd "[:alpha:],[]'")
89+
targets=$(cat artifacts/matrix_info/targets.txt | tr -cd "[:alnum:],[]'")
90+
91+
echo "base = $base"
92+
echo "targets = $targets"
93+
echo "types = $types"
7294
echo "pr_num = $pr_num"
7395
7496
printf "$ref" >> artifacts/ref.txt
@@ -98,6 +120,9 @@ jobs:
98120
cat artifacts/conclusion.txt
99121
100122
echo "pr_num=$pr_num" >> $GITHUB_OUTPUT
123+
echo "base=$base" >> $GITHUB_OUTPUT
124+
echo "targets=$targets" >> $GITHUB_OUTPUT
125+
echo "types=$types" >> $GITHUB_OUTPUT
101126
echo "ref=$ref" >> $GITHUB_OUTPUT
102127
103128
- name: Download and extract parent hardware results
@@ -164,8 +189,8 @@ jobs:
164189
strategy:
165190
fail-fast: false
166191
matrix:
167-
type: ['validation']
168-
chip: ['esp32', 'esp32s2', 'esp32s3', 'esp32c3', 'esp32c6', 'esp32h2']
192+
type: ${{ fromJson(needs.get-artifacts.outputs.types) }}
193+
chip: ${{ fromJson(needs.get-artifacts.outputs.targets) }}
169194
steps:
170195
- name: Report pending
171196
uses: actions/github-script@v7
@@ -211,9 +236,12 @@ jobs:
211236
echo "enabled=$enabled" >> $GITHUB_OUTPUT
212237
213238
# Note that changes to the workflows and tests will only be picked up after the PR is merged
239+
# DO NOT CHECKOUT THE USER'S REPOSITORY IN THIS WORKFLOW. IT HAS HIGH SECURITY RISKS.
214240
- name: Checkout repository
215241
if: ${{ steps.check-tests.outputs.enabled == 'true' }}
216242
uses: actions/checkout@v4
243+
with:
244+
ref: ${{ needs.get-artifacts.outputs.base || github.ref }}
217245

218246
- uses: actions/setup-python@v5
219247
if: ${{ steps.check-tests.outputs.enabled == 'true' }}

‎tests/performance/coremark/coremark.ino

Copy file name to clipboardExpand all lines: tests/performance/coremark/coremark.ino
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ void setup() {
4040
Serial.printf("Cores: %d\n", CONFIG_SOC_CPU_CORES_NUM);
4141
Serial.flush();
4242
for (int i = 0; i < N_RUNS; i++) {
43-
Serial.printf("Run %d", i);
43+
Serial.printf("Run %d\n", i);
4444
coremark_main();
4545
Serial.flush();
4646
}

‎tests/performance/fibonacci/fibonacci.ino

Copy file name to clipboardExpand all lines: tests/performance/fibonacci/fibonacci.ino
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ void setup() {
3131
Serial.printf("N: %d\n", FIB_N);
3232
Serial.flush();
3333
for (int i = 0; i < N_RUNS; i++) {
34-
Serial.printf("Run %d", i);
34+
Serial.printf("Run %d\n", i);
3535
unsigned long start = millis();
3636
fibonacci = fib(FIB_N);
3737
unsigned long elapsed = millis() - start;

‎tests/performance/fibonacci/test_fibonacci.py

Copy file name to clipboardExpand all lines: tests/performance/fibonacci/test_fibonacci.py
+17-16Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,21 @@
22
import logging
33
import os
44

5+
fib_results = {}
6+
7+
def fib(n):
8+
if n < 2:
9+
return n
10+
elif str(n) in fib_results:
11+
return fib_results[str(n)]
12+
else:
13+
fib_results[str(n)] = fib(n - 1) + fib(n - 2)
14+
return fib_results[str(n)]
15+
516

617
def test_fibonacci(dut, request):
718
LOGGER = logging.getLogger(__name__)
819

9-
# Fibonacci results starting from fib(35) to fib(45)
10-
fib_results = [
11-
9227465,
12-
14930352,
13-
24157817,
14-
39088169,
15-
63245986,
16-
102334155,
17-
165580141,
18-
267914296,
19-
433494437,
20-
701408733,
21-
]
22-
2320
# Match "Runs: %d"
2421
res = dut.expect(r"Runs: (\d+)", timeout=60)
2522
runs = int(res.group(0).decode("utf-8").split(" ")[1])
@@ -30,7 +27,11 @@ def test_fibonacci(dut, request):
3027
res = dut.expect(r"N: (\d+)", timeout=300)
3128
fib_n = int(res.group(0).decode("utf-8").split(" ")[1])
3229
LOGGER.info("Calculating Fibonacci({})".format(fib_n))
33-
assert fib_n > 30 and fib_n < 50, "Invalid Fibonacci number"
30+
assert fib_n > 0, "Invalid Fibonacci number"
31+
32+
# Calculate Fibonacci results
33+
expected_result = fib(fib_n)
34+
LOGGER.info("Expected Fibonacci result: {}".format(expected_result))
3435

3536
list_time = []
3637

@@ -48,7 +49,7 @@ def test_fibonacci(dut, request):
4849
assert fib_result > 0, "Invalid Fibonacci result"
4950

5051
# Check if the result is correct
51-
assert fib_result == fib_results[fib_n - 35]
52+
assert fib_result == expected_result
5253

5354
# Match "Time: %lu.%03lu s"
5455
res = dut.expect(r"Time: (\d+)\.(\d+) s", timeout=300)

‎tests/performance/psramspeed/ci.json

Copy file name to clipboardExpand all lines: tests/performance/psramspeed/ci.json
+3-5Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,7 @@
33
"qemu": false,
44
"wokwi": false
55
},
6-
"targets": {
7-
"esp32c3": false,
8-
"esp32c6": false,
9-
"esp32h2": false
10-
}
6+
"requires": [
7+
"CONFIG_SPIRAM=y"
8+
]
119
}

‎tests/performance/psramspeed/psramspeed.ino

Copy file name to clipboardExpand all lines: tests/performance/psramspeed/psramspeed.ino
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,7 @@ void setup() {
252252
Serial.printf("Max test size: %d\n", MAX_TEST_SIZE);
253253
Serial.flush();
254254
for (int i = 0; i < N_RUNS; i++) {
255-
Serial.printf("Run %d", i);
255+
Serial.printf("Run %d\n", i);
256256
memcpy_speed_test(dest, src, MAX_TEST_SIZE, N_COPIES);
257257
Serial.flush();
258258
memset_speed_test(dest, FILL_VALUE, MAX_TEST_SIZE, N_COPIES);

‎tests/performance/ramspeed/ramspeed.ino

Copy file name to clipboardExpand all lines: tests/performance/ramspeed/ramspeed.ino
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -248,7 +248,7 @@ void setup() {
248248
Serial.printf("Max test size: %d\n", MAX_TEST_SIZE);
249249
Serial.flush();
250250
for (int i = 0; i < N_RUNS; i++) {
251-
Serial.printf("Run %d", i);
251+
Serial.printf("Run %d\n", i);
252252
memcpy_speed_test(dest, src, MAX_TEST_SIZE, N_COPIES);
253253
Serial.flush();
254254
memset_speed_test(dest, FILL_VALUE, MAX_TEST_SIZE, N_COPIES);

‎tests/performance/superpi/superpi.ino

Copy file name to clipboardExpand all lines: tests/performance/superpi/superpi.ino
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ void setup() {
2525
Serial.printf("Digits: %d\n", DIGITS);
2626
Serial.flush();
2727
for (int i = 0; i < N_RUNS; i++) {
28-
Serial.printf("Run %d", i);
28+
Serial.printf("Run %d\n", i);
2929
unsigned long start = millis();
3030
pi_calc(DIGITS);
3131
unsigned long elapsed = millis() - start;

‎tests/requirements.txt

Copy file name to clipboard
+4-4Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
cryptography==43.0.1
22
--only-binary cryptography
33
pytest-cov==5.0.0
4-
pytest-embedded-serial-esp==1.11.5
5-
pytest-embedded-arduino==1.11.5
6-
pytest-embedded-wokwi==1.11.5
7-
pytest-embedded-qemu==1.11.5
4+
pytest-embedded-serial-esp==1.11.6
5+
pytest-embedded-arduino==1.11.6
6+
pytest-embedded-wokwi==1.11.6
7+
pytest-embedded-qemu==1.11.6

‎tests/validation/gpio/test_gpio.py

Copy file name to clipboard
+10Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
1+
import logging
2+
13
def test_gpio(dut):
4+
LOGGER = logging.getLogger(__name__)
5+
26
dut.expect_exact("Button test")
7+
8+
LOGGER.info("Expecting button press 1")
39
dut.expect_exact("Button pressed 1 times")
10+
11+
LOGGER.info("Expecting button press 2")
412
dut.expect_exact("Button pressed 2 times")
13+
14+
LOGGER.info("Expecting button press 3")
515
dut.expect_exact("Button pressed 3 times")

‎tests/validation/nvs/test_nvs.py

Copy file name to clipboard
+9Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,13 @@
1+
import logging
2+
13
def test_nvs(dut):
4+
LOGGER = logging.getLogger(__name__)
5+
6+
LOGGER.info("Expecting counter value 0")
27
dut.expect_exact("Current counter value: 0")
8+
9+
LOGGER.info("Expecting counter value 1")
310
dut.expect_exact("Current counter value: 1")
11+
12+
LOGGER.info("Expecting counter value 2")
413
dut.expect_exact("Current counter value: 2")

‎tests/validation/periman/test_periman.py

Copy file name to clipboardExpand all lines: tests/validation/periman/test_periman.py
+6-2Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
1+
import logging
2+
13
def test_periman(dut):
4+
LOGGER = logging.getLogger(__name__)
25
peripherals = [
36
"GPIO",
47
"SigmaDelta",
@@ -29,9 +32,10 @@ def test_periman(dut):
2932

3033
if peripheral in peripherals:
3134
if "not" in console_output:
32-
assert False, f"Peripheral {peripheral} printed when it should not"
35+
assert False, f"Output printed when it should not after peripheral {peripheral}"
36+
LOGGER.info(f"Correct output after peripheral: {peripheral}")
3337
peripherals.remove(peripheral)
3438
else:
3539
assert False, f"Unknown peripheral: {peripheral}"
3640

37-
assert peripherals == [], f"Missing peripherals output: {peripherals}"
41+
assert peripherals == [], f"Missing output after peripherals: {peripherals}"

‎tests/validation/psram/ci.json

Copy file name to clipboardExpand all lines: tests/validation/psram/ci.json
+1-6Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,5 @@
55
},
66
"requires": [
77
"CONFIG_SPIRAM=y"
8-
],
9-
"targets": {
10-
"esp32c3": false,
11-
"esp32c6": false,
12-
"esp32h2": false
13-
}
8+
]
149
}

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.