Enforce 100% test coverage

[jtmiclat]

Resolves: #316
Caveats: Currently failing as test coverage is not 100%

[sethmlarson]

@jtmiclat Could you make coverage only apply to Python 3.7+? 3.6 is deliberately missing a large chunk of coverage.

[JayH5]

Is it not possible to get Codecov to enforce coverage? In the past when I have used it it shows up as a CI check next to Travis but maybe it's not configured for that at the moment?

[sethmlarson]

Might require @tomchristie to add a codecov to the org somehow, that's how I'd prefer to handle this as well.

[jtmiclat]

@sethmlarson Just added skipping 100% enforcement of test coverage for python 3.6!

[lovelydinosaur]

I've generally switched to enforcing it alongside the tests because I found that otherwise we ended up with a large number of pull requests that didn't keep 100% coverage because devs were missing it at the point of commit.

Ideally I'd have ./scripts/test be a comprehensive "this also oughta pass CI", so that we don't end up with lots of churn of "commit, PR, CI fail, try again".

[florimondmanca]

I personally enjoy Codecov reports as well for their detailed output and visualization, but agree with Tom that we should detect insufficient coverage as early as possible. Having the tests fail locally in that case is a good way to get that feedback.

So... Can we just use both?

[sethmlarson]

Btw I'm working on improving code coverage currently, will submit that change then we can merge this. :)

[florimondmanca]

Doing some cleanup, and I think we're not just ready to turn on 100% coverage yet — still some small additions needed to reach it. I'll close this but we can revisit and see how we'd like to turn on 100% checks later, yes?