Evy is tested using Nose. To run tests, simply install nose, and then, in the evy tree, do:
$ python setup.py test
If you want access to all the nose plugins via command line, you can run:
$ python setup.py nosetests
Lastly, you can just use nose directly if you want:
$ nosetests
That’s it! The output from running nose is the same as unittest’s output, if the entire directory was one big test file.
Many tests are skipped based on environmental factors. These are printed as S’s during execution, and in the summary printed after the tests run it will tell you how many were skipped.
To run the doctests included in many of the evy modules, use this command:
$ nosetests --with-doctest evy/*.py
Currently there are 16 doctests.
Evy provides for the ability to test itself with the standard Python networking tests. This verifies that the libraries it wraps work at least as well as the standard ones do. The directory tests/stdlib contains a bunch of stubs that import the standard lib tests from your system and run them. If you do not have any tests in your python distribution, they’ll simply fail to import.
There’s a convenience module called all.py designed to handle the impedance mismatch between Nose and the standard tests:
$ nosetests tests/stdlib/all.py
That will run all the tests, though the output will be a little weird because it will look like Nose is running about 20 tests, each of which consists of a bunch of sub-tests. Not all test modules are present in all versions of Python, so there will be an occasional printout of “Not importing %s, it doesn’t exist in this installation/version of Python”.
If you see “Ran 0 tests in 0.001s”, it means that your Python installation lacks its own tests. This is usually the case for Linux distributions. One way to get the missing tests is to download a source tarball (of the same version you have installed on your system!) and copy its Lib/test directory into the correct place on your PYTHONPATH.
What follows are some notes on writing tests, in no particular order.
The filename convention when writing a test for module foo is to name the test test_foo.py. We don’t yet have a convention for tests that are of finer granularity, but a sensible one might be test_foo_class.py.
If you are writing a test that involves a client connecting to a spawned server, it is best to not use a hardcoded port because that makes it harder to parallelize tests. Instead bind the server to 0, and then look up its port when connecting the client, like this:
from evy.io.convenience import listener, connect
server_sock = listener(('127.0.0.1', 0))
client_sock = connect(('localhost', server_sock.getsockname()[1]))
Coverage.py is an awesome tool for evaluating how much code was exercised by unit tests. Nose supports it if both are installed, so it’s easy to generate coverage reports for evy. Here’s how:
nosetests --with-coverage --cover-package=evy
After running the tests to completion, this will emit a huge wodge of module names and line numbers. For some reason, the --cover-inclusive option breaks everything rather than serving its purpose of limiting the coverage to the local files, so don’t use that.
The html option is quite useful because it generates nicely-formatted HTML that are much easier to read than line-number soup. Here’s a command that generates the annotation, dumping the html files into a directory called “cover”:
coverage html -d cover --omit='tempmod,<console>,tests'
(tempmod and console are omitted because they gets thrown away at the completion of their unit tests and coverage.py isn’t smart enough to detect this.)