Speed up testpythoneval

[ambv]

Splits eval-test into simple buckets by first letter of test name, enabling
parallel execution. This speeds up execution of the test suite by around 25% on
my laptop. The split enables more consistent loading of all CPU cores during
the entire run of ./runtests.py.

To achieve this, I had to modify testpythoneval.py to not write all testcase
inputs to the same temporary path.

Before:

  SUMMARY  all 204 tasks and 1811 tests passed
  *** OK ***
  total time in run: 554.571954
  total time in check: 214.105742
  total time in lint: 130.914682
  total time in pytest: 92.031659
  ./runtests.py -j4 -v  744.76s user 74.10s system 235% cpu 5:48.34 total

After:

  SUMMARY  all 225 tasks and 3823 tests passed
  *** OK ***
  total time in run: 640.698327
  total time in check: 178.758370
  total time in lint: 149.604402
  total time in pytest: 78.356671
  ./runtests.py -j4 -v  850.81s user 81.09s system 353% cpu 4:23.69 total

Total wall clock time fell from 5:48 to 4:23.

Note: the test sum is now over-reported. Looks like the driver counts also the
filtered out tests in eval-test. I don't have cycles now to hunt this down.

[ambv]

While I wouldn't bet on Travis for benchmarking, master was consistently taking around 20 minutes to pass all tests. With this change, this is down to 11 minutes. ✨

[ambv]

OK, this is now much more friendly for review. Under 100 significant lines in total, just two files.

[gvanrossum]

Looks good! Thanks for doing this. (But really, I wish we could kill runtests.py in favor of migrating everything to pytest...)

[ambv]

(But really, I wish we could kill runtests.py in favor of migrating everything to pytest...)

Agreed! I see there's discussion on #1673, let me comment there on steps forward. For this pull request here, parallelizing testpythoneval.py was the lowest hanging fruit in terms of runtime savings which cuts down our waits today :)