Why should django.template be refactored?
Filter and Variable parsing is inconsistent. Many ad hoc parsers in defaulttags are fragile.
Whitespace handling is ungraceful.
The patch provided splits __init__.py into separate modules and introduces a TokenStream
class that allows parsing of literals, vars (called lookups) and filter expressions. No docs yet,
have a look at the source (it all happens in ~150 loc).
The patch only touches django.template and django.templatetags.
Tests are separate so you can run them with the old code.
Modules in django.template
- context (no dependencies)
Untouched.
- expressions (no dependencies)
Contains Lookup, Literal, and FilterExpression. All are subclasses of a marker
class Expression. These replace Variable and the old FilterExpression class.
A Variable equivalent is provided in compat.Variable.
- nodes (depends on expressions)
Contains Node, NodeList, TextNode, and ExpressionNode. The latter
is the old VariableNode renamed. A VariableNode alias is provided in compat.
- compiler (depends on context, expressions, nodes)
Contains Parser, Lexer, Token, TemplateSyntaxError,
Origin, StringOrgin, and compile_string(). Those are mostly untouched.
Additionally there are TokenStream, and TokenSyntaxError. Those provide
a safe way to parse Tokens containing expressions.
- library (depends on compiler, nodes, context)
Contains InvalidTemplateLibrary, Library, get_library(), and add_to_builtins().
Mostly untouched. Should be called libraries for consistency, but that would clash
with the legacy dict.
- loader
Untouched. Moved TemplateDoesNotExist here.
- utils
ConditionalNode, base class for IfNode, IfEqualNode. EmptyNode.
Helper functions:
parse_conditional_nodelists(parser, name), parse_as(bits)
parse_args_and_kwargs(bits), resolve_args_and_kwargs(args, kwargs)
- compat
Provides backwards compatibility for Variable, VariableNode, TokenParser,
and resolve_variable()
- defaulttags, loader_tags
Mostly refactored to use TokenStream where appropriate.
Tickets
- #4746 - allow whitespace before and after filter separator
fixed. tests are there (filter-syntax03, filter-syntax04) but expect TemplateSyntaxError?.
- #5270 - empty string literals
fixed. tests included.
- #5756 - accept filters (almost) everywhere
fixed. a few tests included.
- #5971 - token parser bug
TokenParser will be deprecated.
- #6271 - filter arguments with spaces
fixed. test included. (seems to be fixed in trunk already)
- #6296 - expressions for ifequal
fixed. tests included (dup of #5756 ?)
- #6510 - get_nodes_by_type() problem
probably fixed.
- #6535 - negative numeric literals
fixed. tests included.
- #7295 - quotes, escaping and translation of string literals handled inconsistently in templates
fixed.
Comparing runtests.py templates measures, the refactored version is a bit slower.
Todo
{% if not %} is currently valid. The refactored implementation is greedy and reports a syntax error.
Could be fixed, not sure it's worth it. Broken tests are: if-tag-not02, if-tag-not03. DDN.
Numeric literals ending with "." are now considered a syntax error.
Broken test: ifequal-numeric07. DDN.
{% url %} currently accepts an arbitrary unquoted unicode string for the view name.
The refactored implementation only accepts bare python identifiers or quoted string literals.
Broken test: url05. DDN.
TemplateSyntaxError messages need some work. Eventually shortcut methods on TokenStream.
Find better names for TokenSyntaxError and TokenStream.
Provide a decorator that handles the boilerplate bits=parser.token_stream(token)
and bits.assert_consumed() calls.
Investigate performance impact.
Docs and more tests.
