Move the core of the combine logic to be entirely in SQL#2033
Move the core of the combine logic to be entirely in SQL#2033nedbat merged 1 commit intocoveragepy:mastercoveragepy/coveragepy:masterfrom
Conversation
|
Also a few tests are failing 😂. Before I run too much farther on this path, wanted to see how you feel about it? |
|
It's an interesting idea, thanks for starting it. What in-memory databases are you concerned about? |
|
|
|
Right, i know the API makes it possible, I'm just wondering realistically when it happens. |
|
Oh, I have no idea. I just assumed I wasn't allowed to break it either way :-) I think there's a way I can make it work though. |
378c05b to
5da41b8
Compare
|
This should be ready for review now, all green :-) |
|
Looking at a few recent runs, this seems to make my combine step slightly slower, though these are hardly precise measurements (with this: 2m 57s, 2m 59s), (without this: 2m 35s, 2m 48s). How many files are you combining, and what are the times? |
|
29 .coverage files + 30 lcov files (that I pre-merge into a single On current I'm guessing you must have a lot more data for it to nearly 3 minutes? I can profile (and maybe do some SQL |
|
With some quick instrumentation, I can see that ~85% of my time is being spent merge the arcs data, are you using branch coverage for yours or line? |
|
I am using branch coverage, but I was also quoting the time for my "combine and report" step. Splitting it into parts shows that combine is not the problem: I have to assume my timing difference is noise. (the coverage.json is 2.8Gb, and I don't even keep it from CI!) |
This shaves roughly 40% of the runtime off of pyca/cryptography's combine. A major limitation is that it currently doesn't work with in-memory sqlite databases, I'm not sure if there's a good way to do that.
5da41b8 to
270ed94
Compare
|
Well, investigating I found another ~2-2.5 seconds to save, I'm now down to 7s consistently. |
|
It's good: before this: 3.59s, 3.56s. with this: 1.47s, 1.51s. Excellent |
|
Perfect! |
|
This was amazing, thanks so much! |
|
Thank you! |
|
This is now released as part of coverage 7.10.5. |
|
Awesome, thank you!
All that is necessary for evil to succeed is for good people to do nothing.
…On Sat, Aug 23, 2025, 10:45 AM Ned Batchelder ***@***.***> wrote:
*nedbat* left a comment (coveragepy/coveragepy#2033)
<#2033 (comment)>
This is now released as part of coverage 7.10.5
<https://pypi.org/project/coverage/7.10.5>.
—
Reply to this email directly, view it on GitHub
<#2033 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBDEOOJAQJL67RRX5H33PB47PAVCNFSM6AAAAACEDEXCTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJXGA3TEOBRGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
This shaves roughly 40% of the runtime off of pyca/cryptography's combine.
A major limitation is that it currently doesn't work with in-memory sqlite databases, I'm not sure if there's a good way to do that.