Testing¶
Tests are central to our development process and hence some understanding of how testing is done will also help you to understand Reahl – and the rest of this tutorial.
Here’s how we like to think of tests: if you had to explain a system to a newcomer from scratch, you’d explain a number of facts – some facts also build upon on other facts that have already been explained. We try to identify these explanatory facts, and write a test per fact. The code of the test itself provides an example and/or more detail about that fact. Tests are thus a specification of the requirements of the system, and can be used to explain the system to someone.
In order to facilitate these ideas, some testing tools grew with Reahl. This section gives a brief introduction.
Fixtures¶
In order to test something, you usually need to create a number of related objects first for use in the test itself. Creating these objects can be tedious – especially if they depend on each other.
Such a collection of objects used together by a test is called a test
Fixture. Reahl allows one to write a Fixture as a separate
class. Defining a Fixture separately from the test that is using it
allows for the Fixture to be re-used by different tests, and even by
other Fixture classes. Separating the Fixture also helps keep the test
itself to read more like a requirements specification and less like
code.
Here’s how Fixtures work: For each object that will form part of the
Fixture, a programmer writes a method on the Fixture class which will
create the object when called. The name of the method is “new_”
prefixed to the name of the object. (Assuming an object named
my_object, you would add a method to the fixture named
new_my_object.) In order to use a Fixture with a specific test,
decorate the test using the @test() decorator, passing the Fixture
class to @test(). The test method or function should also
expect a single argument in its signature: the Fixture instance.
The first time a program references .my_object on the Fixture
instance, the corresponding new_ method will be called behind the
scenes to create the object. Subsequent accesses of .my_object will
always bring back the same instance which was created on the first
access.
Here is a simple test which illustrates how Fixtures work:
from __future__ import print_function, unicode_literals, absolute_import, division
from reahl.tofu import test, Fixture
class SimpleFixture(Fixture):
def new_name(self):
return 'John'
@test(SimpleFixture)
def fixture_singletons(fixture):
"""Accessing an attribute on the Fixture always brings back the same
object, as created by a similarly named new_ method on the fixture."""
assert fixture.name == 'John'
assert fixture.name is fixture.name # ie, the same object is always returned
Fixtures are all about objects that depend on one another. The
following example shows a Fixture with two objects: a .name, and a
.user. The User object is initialised using the .name which
accompanies it on the Fixture. For simplicity, the actual code of User
is just included in the test. Note also how SimpleFixture is reused
here, by deriving from it:
class User(object):
def __init__(self, name):
self.name = name
class InterestingFixture(SimpleFixture):
def new_user(self):
return User(self.name)
@test(InterestingFixture)
def dependent_setup_objects(fixture):
"""Different attributes on a Fixture can reference one another."""
assert fixture.user.name is fixture.name
Things can get more interesting though. A useful convention is to
create new_* methods with keyword arguments. This way one can use
the new_* method to create slightly modified versions of the default
object available on the Fixture:
class MoreInterestingFixture(SimpleFixture):
def new_user(self, name=None):
return User(name or self.name)
@test(MoreInterestingFixture)
def bypassing_the_singleton(fixture):
"""new_ methods can be supplied with kwargs in order to create test objects that differ from the default."""
jane = fixture.new_user(name='Jane')
assert jane.name == 'Jane'
assert fixture.user is not jane
other_jane = fixture.new_user(name='Jane')
assert jane is not other_jane
Running the tests:
nosetests testbasics.py
Set_up, tear_down, and run fixtures¶
Since Fixtures mostly consist of a bunch of new_ methods, they
usually do not need traditional methods for settting up a fixture, or
tearing the fixture down afterwards. In some rare cases this is still
needed though. One may want to start a web server before a test, and
stop it afterwards, for example. Any method on a Fixture can be run
before or after the test it is used with. Just annotate the method
with @set_up or @tear_down respectively.
Sometimes such setup can take a long time and would slow down tests if
it happens for each and every test. When testing web applications, for
example, you may want to fire up a browser before the test – something
that takes quite a long time. For this reason Reahl provides an
extension to nosetests to which you can specify a “run Fixture”. A run
Fixture is used for all tests that are run together. It is set up
before all the tests are run, and torn down at the end of all test
runs. Normal Fixtures that are attached to individual tests also have
access to the current run Fixture.
To specify which run Fixture should be used for a test run, use
the --with-run-fixture (or -F) argument to nosetests.
Testing without a real browser¶
Enough talk. Its time for a first test.
In the previous section an application was
developed which lets one add, view and edit Addresses on different
Views. The second version of it used Buttons to navigate to the edit
Views of individual Addresses. This application is starting to get
cumbersome to test by hand each time a change is made to it. It needs
a test. A Test for it will also make it possible to set breakpoints
and investigate its internals while running.
Here is a first stab at a test for it. Sometimes it is useful to write a test per explained fact; other times it is useful to write a test for a little scenario illustrating a “user story”. This test is an example of the latter:
from __future__ import print_function, unicode_literals, absolute_import, division
from reahl.tofu import test
from reahl.web_dev.fixtures import WebFixture
from reahl.webdev.tools import Browser, XPath
from reahl.doc.examples.tutorial.parameterised2.parameterised2 import AddressBookUI
@test(WebFixture)
def adding_an_address(fixture):
"""To add a new address, a user clicks on "Add Address" link on the menu, then supplies the
information for the new address and clicks the Save button. Upon successful addition of the
address, the user is returned to the home page where the new address is now listed."""
browser = Browser(fixture.new_wsgi_app(site_root=AddressBookUI))
browser.open('/')
browser.click(XPath.link_with_text('Add an address'))
actual_title = browser.title
assert actual_title == 'Add an address', 'Expected to be on the Add an address page'
browser.type(XPath.input_labelled('Name'), 'John Doe')
browser.type(XPath.input_labelled('Email'), '[email protected]')
browser.click(XPath.button_labelled('Save'))
actual_title = browser.title
assert actual_title == 'Addresses', 'Expected to be back on the home page after editing'
assert browser.is_element_present(XPath.paragraph_containing('John Doe: [email protected]')), \
'Expected the newly added address to be listed on the home page'
The test should be run with nosetests,
using --with-run-fixture=reahl.webdev.fixtures:BrowserSetup.
There are a number of Fixtures available as part of Reahl to help you
test applications. Explaining all of them is outside of the scope of
this introductory section. For this test, it is useful to know some of
the things that BrowserSetup does: Before the tests start to run, it
creates an empty database with the complete schema of your
application. The Elixir/Sqlalchemy classes used by your application
are also initialised properly (no need to create_all/setup_all, etc).
The test is run with a WebFixture. WebFixture depends on the presence
of a BrowserSetup run Fixture – that’s why BrowserSetup should be
specified to nosetests when run.
The only obvious feature of WebFixture used in this little test is the
creation of a WSGI application. WebFixture.new_wsgi_app() can create a
WSGI application in various different ways. This example shows how to
create a ReahlWSGIApplication from a supplied UserInterface. By default javascript
is not turned on (although, as you will see later, this can be
specified as well).
Behind the scenes, WebFixture also has the job of starting a new
database transaction before each test, and rolling it back after each
test. Hence no real need to tear down anything...
Two other important tools introduced in this test are: Browser and
XPath. Browser is not a real browser – it is our thin wrapper on top
of what is available from WebTest. Browser has the interface of a
Browser though. It has methods like .open() (to open an url),
.click() (to click on anything), and .type() (to type something
into an element of the web page).
We like this interface, because it makes the tests more readable to a non-programmer.
When the browser is instructed to click on something or type into
something, an XPath expression is used to specify how to find that
element on the web page. XPath has handy methods for constructing
XPath expressions while keeping the code readable. Compare, for
example the following two lines to appreciate what XPath does. Isn’t
the one using XPath much more explicit and readable?
browser.click('//a[href="/news"]')
browser.click(XPath.link_labelled('News'))
Readable tests¶
Often in tests, there are small bits of code which would be more
readable if it was extracted into properly named methods (as done with
XPath above). If you create a Fixture specially for testing the
AddressBookUI, such a fixture is an ideal home for such methods. In
the expanded version of our test below, AddressAppFixture was
added. AddressAppFixture now contains the nasty implementation of a
couple of things we’d like to assert about the application, as well as
some objects used by the tests. (Compare this implementation of
.adding_an_address() to the previous implementation.)
from __future__ import print_function, unicode_literals, absolute_import, division
from reahl.tofu import test
from reahl.web_dev.fixtures import WebFixture
from reahl.webdev.tools import Browser, XPath
from reahl.doc.examples.tutorial.parameterised2.parameterised2 import AddressBookUI, Address
class AddressAppFixture(WebFixture):
def new_browser(self):
return Browser(self.new_wsgi_app(site_root=AddressBookUI))
def new_existing_address(self):
address = Address(name='John Doe', email_address='[email protected]')
address.save()
return address
def is_on_home_page(self):
return self.browser.title == 'Addresses'
def is_on_add_page(self):
return self.browser.title == 'Add an address'
def is_on_edit_page_for(self, address):
return self.browser.title == 'Edit %s' % address.name
def address_is_listed_as(self, name, email_address):
return self.browser.is_element_present(XPath.paragraph_containing('%s: %s' % (name, email_address)))
@test(AddressAppFixture)
def adding_an_address(fixture):
"""To add a new address, a user clicks on "Add Address" link on the menu, then supplies the
information for the new address and clicks the Save button. Upon success addition of the
address, the user is returned to the home page where the new address is now listed."""
browser = fixture.browser
browser.open('/')
browser.click(XPath.link_with_text('Add an address'))
assert fixture.is_on_add_page()
browser.type(XPath.input_labelled('Name'), 'John Doe')
browser.type(XPath.input_labelled('Email'), '[email protected]')
browser.click(XPath.button_labelled('Save'))
assert fixture.is_on_home_page()
assert fixture.address_is_listed_as('John Doe', '[email protected]')
@test(AddressAppFixture)
def editing_an_address(fixture):
"""To edit an existing address, a user clicks on the "Edit" button next to the chosen Address
on the "Addresses" page. The user is then taken to an "Edit" View for the chosen Address and
can change the name or email address. Upon clicking the "Update" Button, the user is sent back
to the "Addresses" page where the changes are visible."""
browser = fixture.browser
existing_address = fixture.existing_address
browser.open('/')
browser.click(XPath.button_labelled('Edit'))
assert fixture.is_on_edit_page_for(existing_address)
browser.type(XPath.input_labelled('Name'), 'John Doe-changed')
browser.type(XPath.input_labelled('Email'), '[email protected]')
browser.click(XPath.button_labelled('Update'))
assert fixture.is_on_home_page()
assert fixture.address_is_listed_as('John Doe-changed', '[email protected]')
Testing JavaScript¶
The Browser class used above cannot be used for all tests, since it
cannot execute javascript. If you want to test something which makes
use of javascript, you’d need the tests (or part of them) to execute
in something like an actual browser. Doing this requires Selenium, and
the use of the web server started by BrowserSetup for exactly this
eventuality.
DriverBrowser is a class which provides a thin layer over Selenium’s
WebDriver interface. It gives a programmer a similar set of methods
to those provided by the Browser, as used above. An instance of it is
available on the WebFixture via its .driver_browser attribute.
The standard Fixtures that form part of Reahl use Chromium by default
in order to run tests, and they expect the executable for chromium to
be in ‘/usr/lib/chromium-browser/chromium-browser’.
You can change which browser is used by creating a new run Fixture that inherits
from BrowserSetup, and overriding its .web_driver() method. Some
details regarding how the browser is configured (such as the binary
location of the browser) can similarly be changed by overriding the
relevant .new_ method of that class.
When the following is executed, a browser will be fired up, and
Selenium used to test that the validation provided by EmailField works
in javascript as expected. Note also how javascript is enabled for the
web application upon creation:
from __future__ import print_function, unicode_literals, absolute_import, division
from reahl.tofu import test
from reahl.web_dev.fixtures import WebFixture
from reahl.webdev.tools import XPath
from reahl.doc.examples.tutorial.parameterised2.parameterised2 import AddressBookUI, Address
class AddressAppErrorFixture(WebFixture):
def new_wsgi_app(self):
return super(AddressAppErrorFixture, self).new_wsgi_app(site_root=AddressBookUI, enable_js=True)
def new_existing_address(self):
address = Address(name='John Doe', email_address='[email protected]')
address.save()
return address
def error_is_displayed(self, text):
return self.driver_browser.is_element_present(XPath.error_label_containing(text))
@test(AddressAppErrorFixture)
def edit_errors(fixture):
"""Email addresses on the Edit an address page have to be valid email addresses."""
fixture.reahl_server.set_app(fixture.wsgi_app)
browser = fixture.driver_browser
existing_address = fixture.existing_address # Doing this just creates the Address in the database
browser.open('/')
browser.click(XPath.button_labelled('Edit'))
browser.type(XPath.input_labelled('Email'), 'invalid email address')
assert fixture.error_is_displayed('Email should be a valid email address')