Add unit test framework for end-user Django applications

I'm working on a plugin for my test runner, nose (http://somethingaboutorange.com/mrl/projects/nose/) to ease testing of django apps. It works along the lines of the django tests, setting up a test database (or schema) based on the selected settings file and installing the INSTALLED_APPS there, and tearing all of that down at the end of the test run.

It's currently in early alpha -- if you want to check it out, please don't use it with a settings file that points to production data. While I don't believe there are any data-destroying bugs... well, you get the idea.

You can get nose via easy_install:

easy_install nose

And the plugin from svn:

svn co http://svn.nose.python-hosting.com/nose-django/trunk nose-django

Then cd nose-django and install as normal from there using setup.py. Then try nosetests -h and look for the options with 'django' in the name.

My "test views by context" stuff, described on the list

Changes to the django.conf package

Changes to the django.contrib package

Changes to the django.core package

Changes to the modeltests

Changes to the regressiontests

Ok; here's a walkthrough of v1 of the Django testing framework. Note that this is not the end of development; it is just an attempt to set up a generic test framework that is the equal of the existing test/runtests.py. The next step is to implement facilities for fixtures, testing templates, contexts, views, etc.

core

• Added a 'test' target to django-admin
  • Test target gets the settings.TEST_RUNNER method in setings.TEST_MODULE, and invokes it
  • Test method is provided with a list of modules to test, and a verbosity for error reporting.
• Added a '--verbosity/-v' option, which is passed to a few of the targets
• Modified syncdb to allow customized verbosity of output messages.
• Modified syncdb to operate in a 'non-interactive' mode, so you can call syncdb and disable the call to create a superuser
• Modified the parameters of the post_syncdb signal to pass on verbosity and interactivity options.

conf

• Added two new default settings: TEST_MODULE and TEST_RUNNER. These strings identify the module name, and the module symbol that will be used by the django-admin 'test' target to start the test set. Points to the 'simple' test runner by default.

contrib

• Modified the post_syncdb handlers for the contenttypes, auth, and sites applications to cater for the new verbosity and interactivity information.

test

• Moved the copy of doctest.py into the test package.
• simple.py
  • An implementation of a 'simple' testing strategy - i.e., the same strategy currently used by tests/runtests.py
    • Searches for any doctest in the models.py or tests.py module
    • Searches for any unittest in the models.py or tests.py module
    • Composes a unitest.TestSuite? of all these tests
    • Sets up the test db before running the entire suite, and tears it down after running the suite.
  • The simple test runner can also be provided with a list of 'extra' testCases that will be added to the test suite. This is used by the django tests to insert model validation tests.
• testcases.py
  • Some utilites that customize the default behaviour of doctest and unittest.
  • Original intent was to add some TestCase? extensions to this module that implement fixture/db setup type procedures.
• utils.py
  • Methods that other test framework writers might need - specifically, db setup/teardown calls.
  • db setup/teardown use syncdb to setup the database, so all permissions and content types will be in the database during testing (rather than the existing approach that just installs tables.
  • ALL the apps in INSTALLED_APPS are synced, just in case of dependencies between models.

modeltests

• Moved the API_TESTS string to a location that doctest will find by default.
• An alternate approach would be to put this docstring into a 'tests.py' package.

regressiontests

• Moved the API_TESTS string to a location that doctest will find by default.
• Moved each of the othertests as an app within regressiontests. The tests are in the tests.py of each package.
  • If they were simple doctests, they are pretty much unchanged.
  • If they were executing 'assert' tests, they have been ported as unittests
  • Each of the tests.py for an 'othertest' is still a standalone script.

Comments?

The new django.test module (more clean up)

As part of all this, I think #1514 falls into your area of influence (found it during a wander through older bugs). It raises an issue about how to detect/handle exceptions raised internally.

A browser for testing purposes

Here's the next piece of the testing puzzle - a method for testing contexts and templates produced by views.

It acts a little bit like an automatable web browser. The interface allows users to make GET and POST requests; what is returned is a 'test decorated Response' - a normal HttpResponse object, but with extra attributes (specifically, 'context' and 'template') that describe the contexts and templates that were rendered in the process of serving the request.

If a single template was rendered, response.template and response.context will point to the template and context dictionary respectively. If multiple templates were rendered, response.template and response.context will be lists, set such that response.context[n] was used on response.template[n]. If no templates were rendered, response.template and response.context will be None.

Cookies are also handled in a limited fashion, and are preserved for the lifespan of the Browser instance. A simple helper interface exists to assist with submitting to login pages so that you can test @login_required views.

This is all acheived by directly hooking into the WSGI interface, so no server instance is required for the Browser to be used. Template and context details are obtained by listening to a new signal that is emitted whenever the render() method is invoked on a template.

As noted in the patch - this is not intended to reproduce or replace tools such as Twill or Selenium. While these tools are good tests of client behaviour, they cannot check the contents of Context or Template rendering details (except at the level of 'is the rendered output correct').

As noted with the previous patches, the approach here is not to enforce a particular testing framework, but to provide the tools that allow anyone to test any aspect of their Django application.

Here's a sample test session:

from django.test.browser import Browser

# Create a browser
b = Browser()

# Login to the server as 'fred'
assert b.login('/login/',fred,password), "Couldn't log in!"

# GET the page with Widget 3
response = b.get('/widgets/3/')

# Check some response details
assert response.status_code == 200
assert response.context['widget_name'] == 'foo'
assert response.template.name == 'mytemplate.html'
assert '<b>Widget details</b>' in response.content

# POST a new widget
img = open('widget.jpg','r')
post_data = {
  'name': 'bar',
  'size': 3
  'image': img
}
response = b.post('/widgets/new', post_data)
img.close()

# Check that submit page redirected us somewhere
assert response.status_code == 302

Comments?

I should add that browser.patch is independent of the previous patches. This is a continuation of the framework-agnostic approach.

Addendum to first group of patches; revised runtest.py

Combined version of previous patches to django directory, with some minor revisions and cleanup

Combined version of previous patches to tests directory, with some minor revisions and cleanup

Added some revised patches, incorporating some comments that have been received. For ease of applying, they are combined; to allow them to be attached to TRAC, there are two parts. Both the revised-* patches should be applied using patch -p0 from the django trunk directory.

(In [3658]) Refs #2333 - Added test framework. This includes doctest and unittest finders, Django-specific doctest and unittest wrappers, and a pseudo-client that can be used to stimulate and test views.

(In [3659]) Refs #2333 - Added a signal that is emitted whenever a template is rendered, and added a 'name' field to Template to allow easy identification of templates.

(In [3660]) Refs #2333 - Added 'test' target to django-admin script. Includes addition of --verbosity and --noinput options to django-admin, and a new TEST_RUNNER setting to control the tool used to execute tests.

(In [3661]) Refs #2333 - Modified runtests script to use new testing framework. Migrated existing tests to use Django testing framework. All the 'othertests' have been migrated into 'regressiontests', and converted into doctests/unittests, as appropriate.

(In [3666]) Reverted [3659], the 'name' field on Template objects and the signal emitted whenever a template is rendered. Refs #2333.

(In [3689]) Refs #2333 - Added more documentation for testing framework, and clarified some code as a result of trying to describe it.

(In [3706]) Refs #2333 - Added a TEST_DATABASE_NAME setting that can be used to override the 'test_' + DATABASE_NAME naming policy. This setting is then used in runtests.py to restore the use of 'django_test_db' as the Django model/regression test database. Thanks to Michael Radziej for the feedback.

(In [3707]) Refs #2333 - Re-added the template rendering signal for testing purposes; however, the signal is not available during normal operation. It is only added as part of an instrumentation step that occurs during test framework setup. Previous attempt (r3659) was reverted (r3666) due to performance concerns.

(In [3708]) Refs #2333 - Added model test for the test Client.

(In [3709]) Refs #2333 - Removed a call to the signal dispatcher that was mistakenly merged in.

(In [3711]) Refs #2333 - Added documentation for the test Client, and removed a stray import.

(In [3713]) Refs #2333 - Made minor tweaks to the formatting of testing documentation.

(In [3715]) Refs #2333 - Made minor formatting modifications to test framework documentation.

I've been using this framework, and I find it's generally more convenient to actually allow errors during page production to propagate up to the test harness, rather than getting eaten and turned into 500 pages. This can be easily accomplished by changing the big "try" in django.core.handlers.get_response to re-raise exceptions in the final "except" clause.

I'd submit a patch, but this really ought to be a setting of some kind (so you can still test 500 error page generation) and I don't know the best way to do that. Right now I use "PYTHONPATH=~/django_src/ python manage.py test [myapp]" to swap in a Django svn checkout that has this modification hard-coded in whenever I want to do that.

Phase 3 - Fixtures

Phase 3 of the Django testing framework is Fixtures - the ability to set up the test database to contain a specific data so that the test harness doesn't have to manually create and delete data. As an added bonus, the approach taken here allows for fixtures to be used as a crude backup mechanism, or as a crude schema evolution mechanism.

To make fixtures as simple as possible, I have used the serialization framework, and added a mechanism to load files of serialized data into the database. The patch (fixtures.diff) does the following:

• Adds 'installfixture' target to manage.py. A call to './manage.py installfixture foo' will:
  • Look in a 'fixtures' directory under each app directory (similar to the way that initial_sql looks in the directory named sql).
  • Look for a file foo.EXT, where EXT is the file extension appropriate for that serializer (e.g., foo.xml, foo.json). The file extension is the name of the serializer; json fixtures are the default, but any other serializable format is possible using the --format option.
  • Looks through each of the directories named in settings.FIXTURE_DIRS for fixtures named foo.EXT, and installs all of those.
  • All fixture data is added in a single transaction (i.e., single call to installfixture = single transaction). This is to accomodate forward dependencies in fixture data.

• Renames the 'initial_data' target in manage.py to 'custom_sql'. This is to preserve the ability to provide table-modifying statements, but to discourage the use of the target for data insertion.

• Adds a 'flush' target to manage.py. This removes all data from all tables, then executes post_syncdb on all models (so that any autogenerated content is recreated).

• Adds a 'dumpdb' target to manage.py. This allows the developer to dump the contents of the entire DB, or a subset of apps, in a nominated fixture format.

• Adds a django.test.testcases.TestCase?. This is an extension of unittest.TestCase? that flushes the database at the start of each test, then does an automated fixture installation. Sample usage:

  class MyTest(django.test.TestCase):
      fixtures = ['foo','bar']
      def test_feature1(self):
          # proceed as if 'foo' and 'bar' fixtures has been loaded
      def test_feature2(self):
          # proceed as if 'foo' and 'bar' have been loaded;
          # effects of test_feature1 removed from DB.
  

• Adds code to perform a database flush at the start of each doctest. This slows down the testing process s a bit; however, if you have two doctests (or a doctest and a unit test) in the same application, you get inconsistent results depending on the order of test execution. The alternative is to require/expect users to manually call flush at the start of each doctest for those doctests that are likely to experience order-related issues.

• Adds import shortcuts to allow:

     from django.test import TestCase
     from django.test import Client
  

• Adds a unit test for unit testing with fixtures :-)

Design decisions worth discussing

• I've nominated JSON as the default fixture format; mostly because its the most mature serializer that isn't XML :-)

• There is a get_sql_flush implementation on each database backend; this method does the heavy lifting of the manage.py flush target.

• I've migrated test_client to use the fixtures framework rather than the management.py dispatcher hack for setting up initialization data.

Open issues:

• Should syncdb install a fixture with a known name (e.g., post_syncdb)? This would allow for the automated installation of initial data on sync.

• Database flushing is a very database specific thing. As a result, the get_sql_flush implementation is mostly in the database backends:
  • MySQL allows multiple calls to TRUNCATE, because it doesn't have/check constraints
  • SQLite doesn't have a TRUNCATE, but DELETE is fairly fast.
  • Postgres requires that all TRUNCATEd tables with constraint dependencies appear in a single TRUNCATE request. However, this is only available in Postgres 8.1 or later. Postgres also requires that sequences be reset. A solution for earlier versions of Postgres is required.
  • I can't comment on Oracle or MSSQL, as I don't have a test bed. This requires someone with access to these databases to get the 'flush' command working on these platforms.

• There is one additional change that I would recommend, but is not part of the patch - removal of the 'reset' and 'install' targets in manage.py. These two targets take the output of 'get_sql_' calls, and apply them to the database - except that this isn't what is required anymore. With all the syncing, fixturing, and custom SQL, the get_sql_ calls are useful for debugging, but not really useful as actual reset/install targets.

Please direct discussion to the thread on Django-developers.

Phase 3, version 1 of the testing framework - fixtures

Just realized a minor error in my walkthrough for Phase 3. The example TestCase? should read:

class MyTest(django.test.TestCase):
    fixtures = { 
        'json': ['foo','bar']
        'xml': ['whiz']
    }
    def test_feature1(self):
        # proceed as if 'foo' and 'bar' JSON fixtures and 'whiz' XML fixture has been loaded
    def test_feature2(self):
        # proceed as if 'foo' and 'bar' JSON fixtures and 'whiz' XML fixture has been loaded
        # effects of test_feature1 removed from DB.

I've just attached a revised Fixtures patch for public consideration.

The shape of this patch has been shaped by the discussion I have had with Jacob and others on the django-dev mailing list.

Noteworthy changes since the last version:

• manage.py has loaddata, dumpdata and flush targets to load fixtures, dump fixtures, and flush the contents of the database, respectively.

• manage.py loaddata foo.json will look for JSON fixtures named foo in the application fixture directories, the directories named in FIXTURE_DIRS, and in the absolute path (i.e., loaddata /bar/whiz/foo.json will load the specifically named file).

• manage.py loaddata foo will look for any fixture type named foo; if foo.xml and foo.json are found in the same fixture directory, an error is raised, and the fixture installation is rolled back.

• manage.py install has been removed in favour of syncdb; the 'pass sqlall into the database' approach misses all the signals and indexing that syncdb performs, retrofitting install to have these features would be non-trivial, and ultimately would yield nothing more than a subset of the functionality of syncdb

• manage.py sqlinitialdata has been deprecated, with a message directing to the new name sqlcustom. This is a rename to indicate that initial data should not be in SQL format, but in the new fixtures format.

• The django.test.TestCase? unit test base class is used as follows:

  class MyTest(django.test.TestCase):
      fixtures = 'foo', 'bar.json', 'whiz.xml'
  
      def test_features(self):
          pass
  

  Fixture lookup follows the same pattern as loaddata.

The only open issue at this point is database backend compatibility. SQLite, MySQL, and Postgres 8.1+ are covered. Oracle _should_ work AFAIK, but is untested. ADO is completely untested. Postgres 7.x and 8.0 will not work. The only effect of this patch on unsupported databases is that fixtures wont work, and the fixture-based unit tests fail.

Documentation will be forthcoming once this final design has been approved.

Test Fixtures, version 2

Looking really good, Russ. Can you tell me what's broken on Postgres 8.0 and below? If it's a matter of missing system views or whathaveyou I can likely figure out what needs to be done to fix that.

Jacob:

Postgres 7.x doesn't have a TRUNCATE statement; Postgres 8.0 added TRUNCATE, but it has a problem. If Table1 contains references to Table2, the TRUNCATE approach fails because the table contraint rules kick in. I've tried putting the TRUNCATE calls into a transaction, but it didn't seem to help - the constraint rules seemed to kick in during the transaction.

Postgres 8.1 fixed this problem by allowing you to specify multiple tables in a single TRUNCATE statement:

TRUNCATE table1, table2;

which doesn't activate the constraint checker.

Now; I'm not denying that the problem with 8.0 may exist (at least partially) between my keyboard and my chair; I might have just messed up in my use of transactions when I was testing with 8.0. However, this still leaves the Postgres 7.x issue. 7.x will need to regress to either deletion of rows (like the SQLite implementation), a table destroy/resync, or a complete database destroy/recreate. Supporting 7.x also introduces the need to identify what database version is running so that the correct solution can be applied.

This is all made more difficult by the fact that I have very limited access to a Postgres 8.0 install, and no access to a Postgres 7.x install, so my ability to test these earlier versions is somewhat restricted. Any assistance on this front would be greatfully accepted.

Strange, postgresql 7.4 and 7.3 have a TRUNCATE statement, see docs. I'm not sure about earlier releases, but I'd be astonished if it wouldn't be in at least 7.2, too. But it still won't work with foreign key constraints (and this is a strict no, as described in the docs.)

I think that dropping/recreating all tables is not so bad, at least that's something you can easily do for any database. I wouldn't bother but do this if the TRUNCATE fails, and then you don't need a version check. If you don't like it, you could also disable the foreign key constraint temporarily with

ALTER TABLE [ ONLY ] table [ * ]
    DROP CONSTRAINT constraint_name [ RESTRICT | CASCADE ]

and then recreate with

ALTER TABLE [ ONLY ] table [ * ]
    RENAME [ COLUMN ] column TO new_column

TRUNCATE is considered part of the data definition language of sql, like CREATE TABLE, and as such finishes any open transaction before it begins. That's how it can be so much faster than DELETE :-)

Michael:

I stand corrected on TRUNCATE in 7.x. I had a quick look and couldn't find it, but I'll admit that I didn't look that hard once I knew that TRUNCATE in 8.0 wasn't going to work. Thanks for the pointer.

The reason I went with TRUNCATE over DELETE is speed. Fixtures need to be added and removed for every test case in the system, so the fixture installation process needs to be as fast as possible. I initially started with a 'drop constraints and DELETE' approach, but then I remembered the speed advantage in TRUNCATE.

The removal of constraints is a little messy because an implied constraint is created whenever a REFERENCES to a previous table is added, and its a little nasty to find the name of these implied constraints. I'm currently playing with another approach based around putting DEFERRED/DEFERRABLE in the table definitions; I'll let you know how I go.

DEFERRED with DELETE work, but TRUNCATE automatically gets its own transaction, so it doesn't work.

Inserts are *much* faster without foreign key constraints. But most of the time you don't insert a lot of data, or do you?

I'd simply try to use TRUNCATE, and fall back to completely drop all tables and resync if TRUNCATE fails. With a bit of luck, people will improve this for their favourite database and provide patches ;-)

Isn't that a problem with mysql, too? #2720 describes a bug that means that mysql foreign key constraints are created in the wrong way ...

(In [4659]) Fixes #2333 -- Added test fixtures framework.

Replying to russellm:

(In [3706]) Refs #2333 - Added a TEST_DATABASE_NAME setting that can be used to override the 'test_' + DATABASE_NAME naming policy. This setting is then used in runtests.py to restore the use of 'django_test_db' as the Django model/regression test database. Thanks to Michael Radziej for the feedback.

What about TEST_DATABASE_USER and TEST_DATABASE_PASSWORD, etc.? (I'm running on a shared host whose only support for databases is one-per-user.)