Synopsis

pyVows is a behavior driven development framework for Python.

pyVows is inspired by Vows, a BDD framework for node.js.

Much of what's written here is based or the same as in Vows docs.

As its node.js counterpart, pyVows executes your tests in parallel when it makes sense, and sequentially when there are dependencies.

Imagine we are testing a method that sums up two integers, like this:

def test_sum_returns_42():
    result = add_two_numbers(41, 1)

    assert result
    assert int(result)
    assert result == 42
        

Even in a VERY simple scenario like this, we have three assertions in this test. Not too good if we want a single assertion per test, so we could do it like this:

def test_sum_returns_result():
    result = add_two_numbers(41, 1)
    assert result

def test_sum_returns_a_number():
    result = add_two_numbers(41, 1)
    assert int(result)

def test_sum_returns_42():
    result = add_two_numbers(41, 1)
    assert result == 42
        

This is all fine and dandy, except that we are executing the add_two_numbers function three times. In this simple scenario we don't really care if it gets executed many times, but with real code we want to minimize calls so our tests are always as fast as possible.

This is how the above test would look like in pyVows:

class SumContext(Vows.Context):

    def topic(self):
        return add_two_numbers(41, 1)

    def we_get_a_result(self, topic):
        expect(topic).Not.to_be_null()

    def we_get_a_number(self, topic):
        expect(topic).to_be_numeric()

    def we_get_42(self, topic):
        expect(topic).to_equal(42)
        

Don't worry if you don't understand all of it. We'll see it more thoroughly in the next sections.

Here’s another example, this time describing ‘division by zero’:

# division_by_zero_vows.py

from pyvows import Vows, expect

# Create a Test Batch
@Vows.batch
class Divisions(Vows.Context):
    class WhenDividingANumberByZero(Vows.Context):
        def topic(self):
            return 42 / 0

        def we_get_division_by_zero_error(self, topic):
            expect(topic).to_be_an_error_like(ZeroDivisionError)

    class WhenDividingByOne(Vows.Context):
        def topic(self):
            return 42 / 1

        def we_get_the_same_number(self, topic):
            expect(topic).to_equal(42)

        

And run it:

$ pyvows division_by_zero_vows.py

---

And now, a little more involved example—let’s suppose we have a module called ‘the_good_things’, with some fruit in it:

class Strawberry(object):
    def __init__(self):
        self.color = '#ff0000';

    def isTasty(self):
        return True

class PeeledBanana(object): pass

class Banana(object):
    def __init__(self):
        self.color = '#fff333';

    def peel(self):
        return PeeledBanana()
        

Now write some tests in the_good_things_vows.py:

from pyvows import Vows, expect

from the_good_things import Strawberry, Banana, PeeledBanana

@Vows.batch
class TheGoodThings(Vows.Context):
    class AStrawberry(Vows.Context):
        def topic(self):
            return Strawberry()

        def is_red(self, topic):
            expect(topic.color).to_equal('#ff0000')

        def and_tasty(self, topic):
            expect(topic.isTasty()).to_be_true()

    class ABanana(Vows.Context):
        def topic(self):
            return Banana()

        class WhenPeeled(Vows.Context):
            def topic(self, banana):
                return banana.peel()

            def returns_a_peeled_banana(self, topic):
                expect(topic).to_be_instance_of(PeeledBanana)
        

And run them with the test runner:

$ pyvows the_good_things_vows.py

Installing

The easiest way to install pyVows, is via pip, the python package manager, as so:

$ pip install pyvows

Or to upgrade:

$ pip install -U pyvows

Note: If you are on a Debian like system (Ubuntu) and pyvows fails to install, you may need to install these packages (as root):

# apt-get install libxslt-dev libxml2-dev

After installing the packages, try to install pyvows again.

Guide

To understand pyVows, we’re going to start with a general overview of the different components involved in writing tests, and then go through some of them in more detail.

Structure of a test batch

Test batches in pyVows are the largest unit of tests. The convention is to have one test batch per file, and have the batch’s class match the file name. Test batches are created with @Vows.batch decorator.

@Vows.batch
class MyTestVows(Vows.Context):
    pass
        

Tests are added to suites in batches. This is done with the @Vows.batch decorator.

You can have as many batches in a suite as you want.

Batches are contexts, that can in itself contain contexts, which describe different components and states you want to test.

@Vows.batch
class AContext(Vows.Context):
    pass

@Vows.batch
class AnotherContext(Vows.Context):
    pass
        

Contexts are executed in parallel, and they are fully asynchronous. The order in which they finish is therefore undefined.

Contexts usually contain topics and vows, which in combination define your tests.

@Vows.batch
class AContext(Vows.Context):
    def topic(self):
        return "something"

    def i_am_a_vow(self, topic):
        # Test the results of the topic
        

Contexts can contain sub-contexts which get executed as soon as the parent context finishes:

@Vows.batch
class AContext(Vows.Context):
    def topic(self):
        return "something"

    def i_am_a_vow(self, topic):
        # Test the results of the topic

    class SubContext(Vows.Context):
        # Executed when AContext is done
        pass

@Vows.batch
class AnotherContext(Vows.Context):
    # Executed in Parallel to AContext
    pass
        

Summary

» A Suite is a set of one or more batches that pyVows will execute.

» A batch is a context, representing a structure of nested contexts.

» A context is a class with an optional topic, zero or more vows and zero or more sub-contexts.

» A topic is a function that returns a value.

» A vow is a function which receives the topic as an argument, and runs an assertion on it.

With that in mind, we can imagine the following grammar:

Suite   → Batch*
        Batch   → Context*
        Context → Topic? Vow* Context*
        

Here’s an annotated example:

@Vows.batch                                             # Batch
class Array(Vows.Context):                              # Context
    class AnArray(Vows.Context):                        # Sub-Context
        class WithThreeElements(Vows.Context):
            def topic(self):                            # Topic
                return [1, 2, 3]

            def has_length_of_3(self, topic):           # Vow
                expect(topic).to_length(3)              # Assertion

        class WithZeroElements(Vows.Context):           # Sub-Context
            def topic(self):                            # Topic
                return []

            def has_a_length_of_0(self, topic):         # Vow
                expect(topic).to_length(0)              # Assertion

            class WhenPopped(Vows.Context):
                def topic(self, previous_topic):
                    return previous_topic.pop()

                def raises_when_popped(self, topic):
                    expect(topic).to_be_an_error_like(IndexError)
        

How topics work

Vows introduces an incredibly powerful, yet very simple way of writing your tests. pyVows leverages the same approach towards the same goals.

Understanding topics is one of the keys to understanding pyVows. Unlike other testing frameworks, pyVows forces a clear separation between the element which is tested, the topic, and the actual tests, the vows.

Let’s start with a simple example of a context:

class Test42(Vows.Context):
    def topic(self):
        return 42

    def should_be_equal_to_42(self, topic):
        expect(topic).to_equal(42)
        

So this shows us that the value of the topic is passed down to our test function (refered to as a vow from now on) as an argument. Simple enough. Now what if we have multiple vows?

@Vows.batch
class Test42(Vows.Context):
    def topic(self):
        return 42

    def should_be_a_number(self, topic):
        expect(topic).to_be_numeric()

    def should_be_equal_to_42(self, topic):
        expect(topic).to_equal(42)
        

It works as expected, the value is passed down to each vow. Note that the topic function is only run once.

Scope

Sometimes, you might need a parent topic’s value, from inside a child topic. This is easy, because there is a notion of topic scope. Let’s look at an example:

@Vows.batch
class DataStore(Vows.Context):
    def topic(self):
        return DataStore()

    def should_respond_to_get(self, store):
        expect(store.get).to_be_a_function()

    def should_respond_to_put(self, store):
        expect(store.put).to_be_a_function()

    class CallingGet(Vows.Context):
        def topic(self, store):
            return store.get(42)

        def should_return_the_object_with_id_42(self, topic):
            expect(topic.id).toEqual(42)
        

In the example above, the value of the top-level topic is passed as an argument to the inner topic, in the same manner it’s passed to the vows. For clarity, I named both arguments which refer to the outer topic as store.

Note that the scoping isn’t limited to a single level. Consider:

def topic(self, a, b, c):
    # a being the Parent topic
    # b being the Parent of parent topic
    # c being the Parent of parent of parent topic

    # return something

So the parent topics are passed along to each topic function in the certain order: the immediate parent is always the first argument (a), and the outer topics follow (b, then c), like the layers of an onion.

Running a suite

The simplest way to run a test suite, is with the pyvows command:

pyvows vows/my_vows.py

The results will be printed to the console with the default reporter, which is very similar to Vows 'dot-matrix' reporter.

Running larger suites

When your tests become more complex, spanning multiple files, you’re going to need a way to run them as a single entity.

pyVows' test runner, can be used to run multiple test suites at once. To make use of it, just run it passing the directory as the argument, instead of the file, like this:

pyvows vows/

We can also pass options to pyvows. For example, to get pyvows to run only files that end in '_test', pass the --pattern='*_test.py' argument. The reference section has more information on the different options you can pass to it.

Order of execution and parallelism

We talked about how batches and contexts are executed briefly, but it’s now time to delve into it in more detail:

@Vows.batch
class ReadFile(Vows.Context):
    def topic(self):
        return exists('some_file.txt')

    class AfterSuccessfullyReading(Vows.Context):
        def topic(self, exists):
            if exists:
                return open('some_file.txt').read()
            return None

        def if_exists_we_can_read_the_contents(self, topic):
            expect(topic).to_be_like('some string')
        

In the example above, we make use of nested contexts. As you can tell, the result of the parent topic is passed down to its children, as arguments.

This example as a whole is therefore mostly sequential, while remaining asynchronous.

---

Now let’s look at an example which uses parallel tests to check for some devices:

@Vows.batch
class Exists(Vows.Context):
    class StdOut(Vows.Context):
        def topic(self):
            return exists('/dev/stdout')

        def exists(self, topic):
            expect(topic).to_be_true()

    class Tty(Vows.Context):
        def topic(self):
            return exists('/dev/tty')

        def exists(self, topic):
            expect(topic).to_be_true()

    class DevNull(Vows.Context):
        def topic(self):
            return exists('/dev/null')

        def exists(self, topic):
            expect(topic).to_be_true()

So in this case, the tests can finish in any order, and must not rely on each other. The test suite will exit when the last I/O call completes, and the assertions for it are run.

In other words, sibling contexts are executed in parallel, while nested contexts are executed sequentially. Note that this all happens asynchronously, so while some contexts may be waiting for a parent context to finish, sibling contexts can still execute in the meantime.

Context inheritance

pyVows context model allows for some interesting inheritance scenarios. Imagine we have the following method to test:

def add(a, b):
    return a + b
        

We would write some vows as such:

class Add(Vows.Context):
    def topic(self):
        return add(1, 2)

    def should_be_numeric(self, topic):
        expect(topic).to_be_numeric()

    def should_equal_to_three(self, topic):
        expect(topic).to_equal(3)
        

We might be writing redundant tests, but again, this is just an example. Now imagine we want to test for MANY different combinations. It would be very painful to write the exact same contexts and vows over and over just to test different values. Let's try writing the above in a different way:

def addctx(a, b):
    class Context(Vows.Context):
        def topic(self):
            return add(a, b)

        def should_be_numeric(self, topic):
            expect(topic).to_be_numeric()

        def should_equal_to_three(self, topic):
            expect(topic).to_equal(a + b)

class AddEquals3(addctx(1,2)):
    pass

class AddEquals8(addctx(5,3)):
    pass

class AddEquals10(addctx(6,4)):
    pass

# and so on, and so forth
        

In the above test we are taking advantage of the functional nature of Python and returning a dynamic Context class as the result of addctx. This means that every time addctx is called it's returning a different Context that's aware of the values a and b.

Context inheritance can also be a powerful tool to initialize database connections, web servers and any other resources your vows may rely on. A pretty good example of this is the tornado pyvows project by Rafael Carício.

Using generative testing

Generative testing can be a great ally in having clean, lean tests. Let's get back to our old friend, the add two numbers example:

def add(a, b):
    return a + b

class Add(Vows.Context):
    def topic(self):
        return add(1, 2)

    def should_be_numeric(self, topic):
        expect(topic).to_be_numeric()

    def should_equal_to_three(self, topic):
        expect(topic).to_equal(3)
        

Even though we can simplify this a lot by using context inheritance, like demonstrated in the previous section, we'll try something different now. We'll use generative testing.

Generative testing is having a test (or in our case vows and contexts) be executed one or more times, but with different arguments (topics) in each pass.

Let's rewrite the above using generative testing:

def add(a, b):
    return a + b

test_data = [
    (1, 2, 3),
    (2, 5, 7),
    (3, 4, 7),
    (5, 6, 11)
]

class Add(Vows.Context):
    def topic(self):
        for item in test_data:
            a, b, c = item
            yield (add(a, b), c)

    def should_be_numeric(self, topic):
        sum, expected = topic
        expect(sum).to_be_numeric()

    def should_equal_to_expected(self, topic):
        sum, expected = topic
        expect(sum).to_equal(expected)
        

This means that should_be_numeric and should_equal_to_expected will be executed four times each with a tuple of two items: the result of adding a and b, and the expected result.

Let's take this a step further and check against even more scenarios:

 def add(a, b):
    return a + b

a_samples = range(10)
b_samples = range(10)

class Add(Vows.Context):
    class ATopic(Vows.Context):
        def topic(self):
            for a in a_samples:
                yield a

        class BTopic(Vows.Context):
            def topic(self, a):
                for b in b_samples:
                    yield b

            class Sum(Vows.Context):
                def topic(self, b, a):
                    yield (add(a, b), a + b)

                def should_be_numeric(self, topic):
                    sum, expected = topic
                    expect(sum).to_be_numeric()

                def should_equal_to_expected(self, topic):
                    sum, expected = topic
                    expect(sum).to_equal(expected)
        

This way we are verifying against the sum of all combinations of 0 to 9 plus 0 to 9, yet it is still very simple.

Now we can add as many scenarios as we may think of, and won't have to write a single other vow.

Asynchronous Topics

Say you need to test an async method called async_square that takes a number and returns the square of that number, but asynchronously.

Normally, you'd have a problem, since your tests would not wait for the callback (or even pass a callback for that matter). That's where pyvows comes to your rescue.

Testing async topics is as easy as decorating the topic method with a Vows.async_topic decorator.

def async_square(a, callback):
    callback(a * a, kwarg1=True) # imagine this is async code and could take a while

class Square(Vows.Context):
    @Vows.async_topic
    def topic(self, callback):
        async_square(10, callback)

    def should_be_numeric(self, topic):
        expect(topic[0]).to_be_numeric()

    def should_equal_to_expected(self, topic):
        expect(topic[0]).to_equal(100)

    def should_pass_the_true_flag(self, topic):
        expect(topic.kwarg1).to_equal(True)
        expect(topic['kwarg1']).to_equal(True) # does the same as the previous line
        

Notice the callback argument in the topic. When you decorate the topic with async_topic, pyvows adds this argument.

Simply pass this argument as the callback to the method you wish to test and let pyvows take care of the rest.

Assertions

pyVows features an extensible assertion model, with many useful functions, as well as error reporting.

It’s always best to use the more specific assertion functions when testing a value, you’ll get much better error reporting, because your intention is clearer.

Let’s say we have the following array:

ary = [1, 2, 3]
        

and try to assert that it has 5 elements. With the built-in assert, we would do something like this:

assert len(ary) == 5
        

And get the following error:

AssertionError:

Now let’s try that with one of our more specific assertion functions, to_length:

expect(ary).to_length(5);
        

This reports the following error:

Expected topic([1, 2, 3]) to have 5 of length, but it had 3.

Other useful assertion functions bundled with pyVows include to_match, to_be_instance_of, to_include and to_be_empty—head over to the reference to get the full list.

Custom Assertions

Creating new assertions to be used with expect is as simple as using the Vows.create_assertions decorator on a function that gets the topic as first parameter and the expectation as second:

@Vows.create_assertions
def to_be_greater_than(topic, expected):
    return topic > expected
        

Now doing the following expectation:

expect(2).to_be_greater_than(3);
        

Will report:

Expected topic(2) to be greater than 3.

It will also create the corresponding 'not_' assertion:

expect(4).not_to_be_greater_than(3);
        

Will report:

Expected topic(4) not to be greater than 3.

If you need more control over your error message or your assertion doesn't have a corresponding 'not_', you can use the lower level Vows.assertion decorator and raise a VowsAssertionError. There are lots of examples here.

By raising a VowsAssertionError you get the benefit of highlighting the important values when your specs are giving error.

If you still just wanna raise a AssertionError, like the old times, they are supported:

# attention, this is not a recommendation, just an example of what could be done
@Vows.assertion
def to_be_a_positive_integer(topic):
    assert type(topic) == int, "Expected %s to be a positive integer, but it's not even an integer" % (topic,)
    assert topic != 0, "Expected %s to be a positive integer, but it's 0" % (topic,)
    assert topic > 0, "Expected %s to be a positive integer, but it is a negative integer" % (topic,)

@Vows.assertion
def not_to_be_a_positive_integer(topic):
    assert topic <= 0, "Expected %s not to be a positive integer, but it was" % (topic,)
        

It is really recommended to always declare the assertion and the 'not_' assertion (if applied), so they can be used like this:

        expect(5).to_be_a_positive_integer()
        expect(-3).Not.to_be_a_positive_integer()
        

Reference

The runner and assertion modules are documented here.

Test runner

pyvows [FILE, ...] [options]

Running specific tests

$ pyvows test_1.py
$ pyvows tests/

Running all tests in the current or children folders

$ pyvows

---

Options

-p, --pattern	Pattern of files to run as vows. Defaults to "*_vows.py".
-c, --cover	Indicates that coverage of code should be shown. Defaults to True.
-l, --cover_package	Package to verify coverage. May be specified many times. Defaults to all packages.
-o, --cover_omit	Path of file to exclude from coverage. May be specified many times. Defaults to no files.
-t, --cover_threshold	Coverage number below which coverage is considered failing. Defaults to 80.0.
-r, --cover_report	Store the coverage report as the specified file.
-x, --xunit_output	Enable XUnit output.
-f, --xunit_file	Filename of the XUnit output. Defaults to 'pyvows.xml'.
-v	Verbosity. Can be supplied multiple times to increase verbosity. Defaults to -vv.
--no-color	Does not colorize the output.
--help	Show help
--version	Show current installed pyvows' version

Assertion functions

equality

expect(4).to_equal(4)

expect(5).Not.to_equal(4)

similarity

expect("sOmE RandOm     CAse StRiNG").to_be_like('some random case string')

expect(1).to_be_like(1)
expect(1).to_be_like(1.0)
expect(1).to_be_like(long(1))

expect([1, 2, 3]).to_be_like([3, 2, 1])
expect([1, 2, 3]).to_be_like((3, 2, 1))
expect([[1, 2], [3,4]]).to_be_like([4, 3], [2, 1]])

expect({ 'some': 1, 'key': 2 }).to_be_like({ 'key': 2, 'some': 1 })

expect("sOmE RandOm     CAse StRiNG").Not.to_be_like('other string')
expect(1).Not_to_be_like(2)
expect([[1, 2], [3,4]]).Not.to_be_like([4, 4], [2, 1]])
expect({ 'some': 1, 'key': 2 }).Not.to_be_like({ 'key': 3, 'some': 4 })

type

expect(os.path).to_be_a_function()
expect(1).to_be_numeric()

expect("some").Not.to_be_a_function()
expect("some").Not.to_be_numeric()

truth

expect(True).to_be_true()
expect("some").to_be_true()
expect([1, 2, 3]).to_be_true()
expect({ "a": "b" }).to_be_true()
expect(1).to_be_true()

expect(False).to_be_false()
expect(None).to_be_false()
expect("").to_be_false()
expect(0).to_be_false()
expect([]).to_be_false()
expect({}).to_be_false()

None

expect(None).to_be_null()
expect("some").Not.to_be_null()

inclusion

expect([1, 2, 3]).to_include(2)
expect((1, 2, 3)).to_include(2)
expect("123").to_include("2")
expect({ "a": 1, "b": 2, "c": 3}).to_include("b")

expect([1, 3]).Not.to_include(2)

regexp matching

expect('some').to_match(r'^[a-z]+')

expect("Some").Not.to_match(r'^[a-z]+')

length

expect([1, 2, 3]).to_length(3)
expect((1, 2, 3)).to_length(3)
expect("abc").to_length(3)
expect({ "a": 1, "b": 2, "c": 3}).to_length(3)

expect([1]).Not.to_length(3)

emptiness

expect([]).to_be_empty()
expect(tuple()).to_be_empty()
expect({}).to_be_empty()
expect("").to_be_empty()

exceptions

expect(RuntimeError()).to_be_an_error()

expect(RuntimeError()).to_be_an_error_like(RuntimeError)

expect(ValueError("error")).to_have_an_error_message_of("error")

expect("I'm not an error").Not.to_be_an_error()

expect(ValueError()).Not.to_be_an_error_like(RuntimeError)

expect(ValueError("some")).Not.to_have_an_error_message_of("error")

About

Both pyVows and this website are HEAVILY inspired by the work done by Alexis Sellier, more commonly known as cloudhead. He is responsible for toto, LESS, hijs, and a lot more.

More than that, he's responsible for a shift in the way we write tests. We have cleaner, leaner and meaner tests now. And they *ARE* fast.

pyVows was developed by Bernardo Heynemann, and features contribution by Rafael Carício, Fábio Costa and Daniel Truemper.

The design for this website is the work of Alexis Sellier, and this website is intended as a compliment and as recognition of his great work, not as a copy.