Not totally related,
However it seems to me that these 2 lines are also unnecessary?

@alonme Hmmm, I don't think so because those lines refer to non-debug builds.

I realized the title of this PR could be misleading, so I updated it to reflect that it is specifically for debug builds.

Haha, you're right. The double negative hurt my brain. Updating, thanks!

Nice, let's also drop LTO though (like PGO, it doesn't actually give us any useful new coverage and just takes forever).

How do you feel about also removing the four *-unknown-linux-gnu/clang builds while we're at it? Clang is supposed to be ABI-compatible with GCC, so it sort of just seems like we're wasting more time with those builds (and getting Clang to work properly on those builds requires setting the CC env var and another tweak or two that I forget).