[patch] "Cannot resolve keyword ___ into field" error when working with ManyToMany relation

I have the same problem here with latest SVN (rev 2934). It also occures with .count()

Has anybody got a fix?

Problem transcript

Simple Model

Attached transcript and simple model showing the problem with self referential ManyToManyField?

Just another note: The odd thing about this is that when you use the Admin interface, everything works just fine. Very strange.

Patch files

Ok.. Added a patch that fixes the problem for me... Not extensively tested though...

What database are you using? The 'a1.primary_categories.all()' line works fine for me, and I added your model to the m2m_recursive tests and tried it out and it works fine. I've tried using postgres and mysql.

This happens when using sqlite3. I'll try MySQL if I get a chance.

Ok. Verified that my 'models.py' attached example fails with MySQL as well (and the patch fixes the problem)

More interesting info: Django tests run fine without my patch. The m2m_recursive test fails ('idol' part) with my patch. So the patch is NOT recommended. When I add the 'Person' model to my models, run 'manage.py shell' and manually run the m2m_recursive tests, it fails. So perhaps something isn't being included/initialized properly?

I'm seeing the same error, using mysql and a current checkout from SVN:

>>> e.tags.all()
Traceback (most recent call last):
  File "<console>", line 1, in ?
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 88, in __repr__
    return repr(self._get_data())
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 395, in _get_data
    self._result_cache = list(self.iterator())
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 162, in iterator
    select, sql, params = self._get_sql_clause()
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 409, in _get_sql_clause
    tables2, joins2, where2, params2 = self._filters.get_sql(opts)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 540, in get_sql
    tables2, joins2, where2, params2 = val.get_sql(opts)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 589, in get_sql
    return parse_lookup(self.kwargs.items(), opts)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 697, in parse_lookup
    tables2, joins2, where2, params2 = lookup_inner(path, clause, value, opts, opts.db_table, None)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 800, in lookup_inner
    raise TypeError, "Cannot resolve keyword '%s' into field" % name
TypeError: Cannot resolve keyword 'entry' into field
>>> 

This is very strange. From what I know of that exception, is very unlikely to be DB related -- that exception occurs long before any DB calls are made. So I thought maybe it would be Python version -- but i've just tested with Python 2.3, as well as 2.4, with postgres, mysql and sqlite3, and I simply can't reproduce this. I'm guessing it's one of those subtle dict ordering things, or perhaps a mutable default argument type thing, seeing as it doesn't seem to happen with a simple run through of the tests.

I actually noticed lots of mutable default arguments in django.forms, and there are other ones around the place -- such as one in SortedDict, which could easily be relevant. We really need to go through the code and get rid of them all, but there are one or two I'm not sure about -- such as MultiValueDict.__deepcopy__().

Assuming we've followed PEP 8 (which seems to be the case pretty much everywhere), these two commandlines should find most of them:

egrep -r '=\{\}' *
egrep -r '=\[\]' *

I've no idea whether that's actually related, but it's worth sorting out anyway. I'm off to bed now and not likely to be available until Monday, so if someone else wants to do this they won't be treading on my toes :-)

(Fixed formatting in description.)

Here's a very small test-case that demonstrates the problem:

class Item(models.Model):
    name = models.CharField(maxlength=100)

class Collection(models.Model):
    name = models.CharField(maxlength=100)
    items = models.ManyToManyField(Item)

An interactive session:

>>> from myapp.models import Item, Collection
>>> c = Collection(name="A collection")
>>> c.save()
>>> c.items.all()
Traceback (most recent call last):
  File "<console>", line 1, in ?
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 88, in __repr__
    return repr(self._get_data())
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 395, in _get_data
    self._result_cache = list(self.iterator())
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 162, in iterator
    select, sql, params = self._get_sql_clause()
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 409, in _get_sql_clause
    tables2, joins2, where2, params2 = self._filters.get_sql(opts)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 540, in get_sql
    tables2, joins2, where2, params2 = val.get_sql(opts)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 589, in get_sql
    return parse_lookup(self.kwargs.items(), opts)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 697, in parse_lookup
    tables2, joins2, where2, params2 = lookup_inner(path, clause, value, opts, opts.db_table, None)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/django/db/models/query.py", line 800, in lookup_inner
    raise TypeError, "Cannot resolve keyword '%s' into field" % name
TypeError: Cannot resolve keyword 'collection' into field
>>> 

I get this: (excuse the ipython formatting):

In [3]:c = Collection(name="A collection")

In [4]:c.save()

In [5]:c.items.all()
Out[5]:[]

I'm not going to pursue this until I can reproduce it,

I've now fixed the mutable default arguments thing, but given the nature of the test case and that it works fine for me, I'm pretty sure that won't make any difference.

I made Simon's test and the error was reproduced here. I started looking for this because my tag model in my blog wasn't working.

Interestingly, my admin interface is working fine. The bug only seems to occur in my own code and my interactive sessions.

The problem seems to boil down to this:

>>> Collection._meta.get_all_related_objects()
[]
>>> Item._meta.get_all_related_objects()
[]

Compare that to the django.contrib.auth.models.User model, which works fine:

>>> from django.contrib.auth.models import User
>>> User._meta.get_all_related_objects()
[<RelatedObject: message related to user>, <RelatedObject: logentry related to user>]

lukeplant: It would be useful if you could run this on your own install and see what you get.

With regard to Simon's last comment: get_all_related_objects() depends on the INSTALLED_APPS setting, so perhaps you should post your INSTALLED_APPS as well. Maybe this has something to do with people using the User model without having 'django.contrib.auth' in their INSTALLED_APPS?

INSTALLE D_APPS = (
    'django.contrib.auth',
    'django.contrib.admin',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'simonwillisonnet.bugdemo',
)

Looks OK to me (the space in INTALLE D_APPS needed to get around the Trac spam filter).

Fix problem when running with the management shell

Ok.. It looks like multiple 'Options' objects were getting created. Once when the model was imported and again when a person object was retrieved (it called get_models() in django.db.models.loading package.

When the get_all_related_many_to_many_objects() was called in django.db.models.options package, the test 'self == f.rel.to._meta' (line: 155) failed, so the relation wasn't setup properly.

The above patch calls the get_models() before starting the interpreter so there aren't any inconsistencies.

Curtis: brilliant work! That patch fixes my problem. That said, a more robust solution would probably be to set up Options so that instances representing the same collection of options can be safely compared.

Multiple 'Option' objects per model class shouldn't be created -- this is the bug. I've seen the exact same thing before:

http://groups.google.com/group/django-developers/tree/browse_frm/thread/8fb80e97076e707a/0059cbddcee8eb2c?rnum=1&_done=%2Fgroup%2Fdjango-developers%2Fbrowse_frm%2Fthread%2F8fb80e97076e707a%2F0059cbddcee8eb2c%3Flnk%3Dst%26rnum%3D1%26#doc_02ef913ed50a6705

Basically it seems there is some bug in Python's import statement, or something wrong with the way Django is using it, that causes a module to be reloaded. There are a lot of usages of import(), and I think we need to get this sorted out.

Simon: Thanks. Agreed. BTW, Simon, Luke: Thanks for the quick following ups.

I just noticed something interesting in that get_models() was importing the model into a different module (and apparently the correct one).

For e.g. I was doing from 'mytest.models import *' from the shell. However, get_models() was importing it into 'mysite.mytest.models'.

Trying 'from mysite.mytest.models import *' from the shell caused everything to work correctly.

So could this be more of a problem with the way django sets the sys.path than an import problem?

(In [3202]) Fixed #1796 -- only load a single copy of each model, even when it is referenced via different import paths. Solves a problem with many-to-many relations being set up incorrectly. Thanks to Curtis Thompson, Luke Plant and Simon Willison for some excellent debugging of the problem. Refs #2232 (may also have fixed that).

I can still see this bug when importing models without project name in the import path:

from app.models import Article, Author
...
article.authors.all() # Exception

from project.app.models import Article, Author
...
article.authors.all() # Ok

I'm not reopening the ticket since may be I'm just not getting something obvious. Looking at the patch I wonder if there should be abspath() instead of normpath()?

And here's the model for the reference (though it's pretty straightforward):

class Author(models.Model):
  name = models.CharField(maxlength=20)
  
  def __str__(self):
    return self.name
    
class Article(models.Model):
  title = models.CharField(maxlength=20)
  authors = models.ManyToManyField(Author)
  
  def __str__(self):
    return self.title

Not reopening the bug is not going to get it fixed, so let's correct that for a start.

Since I can't repeat this problem here (it is very system-dependent, as we already know), can you print out the values of app_label, model_name, model_dict, fname1 and fname2 in django/db/models/loadings.py::register_models() around line 96 for the first three (just before the "if" test) and line 102 for the latter two (just after fname1 and fname2 are computed). If we do need something like abspath(), that should show it.

Warning: these print statements will produce a block of output for every model in your system (at least, sometimes more than one per model). So format sensibly. And I really only care about the output for the app and models that are involved in the test, not the whole lot.

Here are conditions:

1. A test app with just 2 exact models I described earlier
2. The project is located in /home/maniac/django_test
3. The app is located in /home/maniac/django_test/m2m
4. All tests are made from project's directory as current, runnning ./manage.py shell
5. PYTHONPATH=/usr/share/django (where only Django is located). But AFAIR manage.py adds also '/home/maniac' to sys.path, so importing from django_test works too.
6. Ubuntu 6.06, Python2.4

This is what importing by app name yields:

>>> from m2m.models import Author, Article

app_label: m2m
model_name: author
model_dict: {}
app_label: m2m
model_name: article
model_dict: {'author': <class 'm2m.models.Author'>}

>>> article = Article.objects.all()[0]
>>> article.authors.all()

app_label: m2m
model_name: author
model_dict: {'article': <class 'm2m.models.Article'>, 'author': <class 'm2m.models.Author'>}
fname1: /home/maniac/django_test/m2m/models.pyc
fname2: m2m/models.pyc
app_label: m2m
model_name: article
model_dict: {'article': <class 'm2m.models.Article'>, 'author': <class 'django_test.m2m.models.Author'>}
fname1: /home/maniac/django_test/m2m/models.pyc
fname2: m2m/models.pyc

-- then follows a traceback leading to "TypeError: Cannot resolve keyword 'article' into field"

Here's with importing by project name:

>>> from django_test.m2m.models import Author, Article

app_label: m2m
model_name: author
model_dict: {}
app_label: m2m
model_name: article
model_dict: {'author': <class 'django_test.m2m.models.Author'>}

>>> article = Article.objects.all()[0]
>>> article.authors.all()

-- No output for these two models (just Django's core)
-- A single author from DB shows OK

Then I moved the code in loading.py to use abspath.

Importing by app name:

>>> from m2m.models import Author, Article

app_label: m2m
model_name: author
model_dict: {}
app_label: m2m
model_name: article
model_dict: {'author': <class 'm2m.models.Author'>}

>>> article = Article.objects.all()[0]
>>> article.authors.all()

app_label: m2m
model_name: author
model_dict: {'article': <class 'm2m.models.Article'>, 'author': <class 'm2m.models.Author'>}
fname1: /home/maniac/django_test/m2m/models.pyc
fname2: /home/maniac/django_test/m2m/models.pyc
app_label: m2m
model_name: article
model_dict: {'article': <class 'm2m.models.Article'>, 'author': <class 'm2m.models.Author'>}
fname1: /home/maniac/django_test/m2m/models.pyc
fname2: /home/maniac/django_test/m2m/models.pyc
[<Author: Author>]

Looks OK.

Importing by project name didn't changed (i.e. my models aren't printed but Author's list is shown).

Just a quick thought - is there anything to the fact that _app_models isn't declared as a global in get_models, get_model or register_model?

If you add "global _app_models" as the first statement in each of these methods (in django.db.models.loading.py), does the problem resolve itself?

Ivan: thanks for the info. I think you're right, we should use abspath() -- easy to fix. I wasn't see such truncated paths on my system (also Linux), but it's clear in your case. I'll commit a fix shortly.

Russell: The global statement isn't needed because we aren't actually assigning to the global name itself, only to references inside it.

(In [3206]) Fixed #1796 -- implemented more robust normalisation for module filenames before comparing them. Ivan Saglaev found a case where r3202 did not work properly.

I've just found that #2232 is not really fixed as I thought yesterday. I described there what happens.

Man, I'm stupid! I keep closing this because it fixes the forwards case (I think -- and yet #2232 keeps on reappearing). But we have not fixed the 'original' report about reverse lookups and they are caused for slightly different reasons. So apologies to whoever "anonymous" was who reported the original: we have still not fixed your problem.

I am going to keep this ticket open to track reverse relation problems.

(In [3212]) Fixed another path where imports were creating two instances of a model's class. Refs #1796, #2232.

As far as I can see, [3212] should have fixed the last problem here. Closing this now.

(In [3279]) Fixed another problem where we were creating a class twice via two import paths. Self-referential relations (e.g. ForeignKey?('self')) were still slipping through the net. Refs #1796.

Broken again by 3490.

To the anonymous re-opener: Would you like me to use The Force to work out what is breaking, or do you have an example you can share? I can see one case in the code that I've missed, but it would be good to have confirmation that this is where you are being tripped up. Thanks for any help you can provide.

(In [3510]) Refs #1796 -- Fixed a problem when creating models with relations to themselves: more than one instance was being created.

The Force is strong with you. Thanks for fixing it =)

You can some models that didn't work here. Sub_Item.objects.all() gave a nice error.

I am reopening this because I am still getting the "Cannot resolve keyword ... into field" error when working with M2M relations in the python shell (python manage.py shell). I am using the latest trunk (revision 3903). I applied the management.patch and fixes the problem...

Patch updated so it can be applied from the root SVN directory.

I can confirm that this is still a bug. I just got bit by it.

The problem only occurs in the shell, and the patch attached to this ticket resolves the issue.

I have re-diffed the patch from the SVN root, and have attached it as new_management.patch.

Couple of confirmations that this patch works.

I'm going to apply this patch, but purely as a temporary workaround. It won't close the ticket, since the patch doesn't fix the problem; it just hides it. We are wall-papering over the real problem here: why isn't the app cache being initialised properly in the case under consideration? If we see the problem in one scenario, who's to say we don't have it occuring elsewhere.

Since the app-cache and model loading code is in the queue for refactoring/redesign in any case, I don't feel to bad about applying a temporary workaround like this. But I'm going to leave the ticket open (and assigned to me).

(In [4533]) Added workaround for loading duplicate model classes in management shell. Patch from Curtis Thompson and Benjamin Slavin. Refs #1796.

i've checked out the latest development version, and still got this error.

I am seeing this problem manifest with current SVN/r4724 both when accessing the admin page for a standard auth_user and interactively from the python shell. The problem appears to be expanding the user.groups relation.

Shell:

>>> from django.contrib.auth.models import User
>>> u = User.objects.get(username__startswith='name')
>>> u
<User: name@example.com>
>>> u.groups
<django.db.models.fields.related.ManyRelatedManager object at 0x6c4490>
>>> u.groups.all()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py", line 102, in __repr__
    return repr(self._get_data())
  File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py", line 468, in _get_data
    self._result_cache = list(self.iterator())
  File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py", line 172, in iterator
    select, sql, params = self._get_sql_clause()
  File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py", line 482, in _get_sql_clause
    joins2, where2, params2 = self._filters.get_sql(opts)
  File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py", line 646, in get_sql
    joins2, where2, params2 = val.get_sql(opts)
  File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py", line 697, in get_sql
    return parse_lookup(self.kwargs.items(), opts)
  File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py", line 829, in parse_lookup
    joins2, where2, params2 = lookup_inner(path, lookup_type, value, opts, opts.db_table, None)
  File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py", line 936, in lookup_inner
    raise TypeError, "Cannot resolve keyword '%s' into field" % name
TypeError: Cannot resolve keyword 'user' into field

Admin:

Traceback (most recent call last):
File "/app/python/2.5.0/lib/python2.5/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/app/python/2.5.0/lib/python2.5/site-packages/django/contrib/admin/views/decorators.py" in _checklogin
  55. return view_func(request, *args, **kwargs)
File "/app/python/2.5.0/lib/python2.5/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  39. response = view_func(request, *args, **kwargs)
File "/app/python/2.5.0/lib/python2.5/site-packages/django/contrib/admin/views/main.py" in change_stage
  363. new_data = manipulator.flatten_data()
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/manipulators.py" in flatten_data
  248. new_data.update(f.flatten_data(fol, obj))
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/fields/related.py" in flatten_data
  700. instance_ids = [instance._get_pk_val() for instance in getattr(obj, self.name).all()]
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py" in __iter__
  108. return iter(self._get_data())
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py" in _get_data
  468. self._result_cache = list(self.iterator())
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py" in iterator
  172. select, sql, params = self._get_sql_clause()
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py" in _get_sql_clause
  482. joins2, where2, params2 = self._filters.get_sql(opts)
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py" in get_sql
  646. joins2, where2, params2 = val.get_sql(opts)
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py" in get_sql
  697. return parse_lookup(self.kwargs.items(), opts)
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py" in parse_lookup
  829. joins2, where2, params2 = lookup_inner(path, lookup_type, value, opts, opts.db_table, None)
File "/app/python/2.5.0/lib/python2.5/site-packages/django/db/models/query.py" in lookup_inner
  936. raise TypeError, "Cannot resolve keyword '%s' into field" % name

  TypeError at /admin/auth/user/2/
  Cannot resolve keyword 'user' into field

Got bitten here to, in my case with Ivan's great tagging app. (Also got a report from a user that adding a User failed at one time.)

Traceback is very similar to the one posted above,

Traceback (most recent call last):
        
          File "/usr/lib/python2.4/site-packages/django/core/handlers/base.py", line 77, in get_response
            response = callback(request, *callback_args, **callback_kwargs)
        
          File "/usr/lib/python2.4/site-packages/django/contrib/admin/views/decorators.py", line 55, in _checklogin
            return view_func(request, *args, **kwargs)
        
          File "/usr/lib/python2.4/site-packages/django/views/decorators/cache.py", line 39, in _wrapped_view_func
            response = view_func(request, *args, **kwargs)
        
          File "/usr/lib/python2.4/site-packages/django/contrib/admin/views/main.py", line 363, in change_stage
            new_data = manipulator.flatten_data()
        
          File "/usr/lib/python2.4/site-packages/django/db/models/manipulators.py", line 248, in flatten_data
            new_data.update(f.flatten_data(fol, obj))
        
          File "/srv/site/src/tags/fields.py", line 77, in flatten_data
            new_data[self.name] = [int(instance.id) for instance in getattr(obj, self.name).all()]
        
          File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 108, in __iter__
            return iter(self._get_data())
        
          File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 468, in _get_data
            self._result_cache = list(self.iterator())
        
          File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 172, in iterator
            select, sql, params = self._get_sql_clause()
        
          File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 482, in _get_sql_clause
            joins2, where2, params2 = self._filters.get_sql(opts)
        
          File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 646, in get_sql
            joins2, where2, params2 = val.get_sql(opts)
        
          File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 697, in get_sql
            return parse_lookup(self.kwargs.items(), opts)
        
          File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 829, in parse_lookup
            joins2, where2, params2 = lookup_inner(path, lookup_type, value, opts, opts.db_table, None)
        
          File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 936, in lookup_inner
            raise TypeError, "Cannot resolve keyword '%s' into field" % name
        
        TypeError: Cannot resolve keyword 'article' into field

In addition to my post of 03/22/07 05:51:20

Think i solved the problem by using full imports everywhere:

from project.tags.models import Tag
from project.app.models import SomeModel

Instead of:

from tags.models import Tag
from app.models import SomeModel

The site have been running trouble free up till now.

No the problem is still there :(

Full imports doesn't solved it. I get the error with the TagField? application in the /admin/ pages during saving.

Could it have something todo with mysite using another subdirectory for custom models ?

Ivan's tags is in "project/tags" while my models (which have tags) are in "project/myapps/article"

#3894 was marked as a duplicate.

#3894 - specifics

Found out why i got this problem, it was not a python 2.3 issue.

I had a middleware that imported a model that also had a reverse m2m to django.contrib.sites.models.Site

Wich fills the self._all_related_many_to_many_objects cache in django/db/models/options.py

So any other model with a reverse m2m is lost, commenting out the line that fills the cache sort of solves it, but im not sure what it does for efficiency.

Øyvind, you're a genius.

This is the link I have been missing in trying to work why we are still not populating the cache correctly. Middleware imports!

Okay, it's still a pain to fix, but now I understand why people are still seeing this and it is impossible to reproduce in my command line tests (they don't import middleware). Thanks for thinking it through. :-)

Add imports in url's to that list, this bug is haunting me. :)

I've uncommented the line Øyvind indicated but same problem for me. It works fine using the shell, (reqroute_proto is the m2m key):

>>> from mt.models import MtReqroute
>>> entry=MtReqroute.objects.get(pk=1)
>>> entry.reqroute_proto.all()
[<MtProto: MGCP>, <MtProto: H.323>]

Now on the server, trying to retrieve the data through a flattened manipulator:

Fails "Cannot resolve keyword 'mtreqroute' into field"

Traceback (most recent call last):
File "/home/yannvr/C4/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/yannvr/C4/django/contrib/auth/decorators.py" in _checklogin
  14. return view_func(request, *args, **kwargs)
File "/var/www/MT/mt/views/routes.py" in edit_route
  54. new_data = manipulator.flatten_data()
File "/home/yannvr/C4/django/db/models/manipulators.py" in flatten_data
  248. new_data.update(f.flatten_data(fol, obj))
File "/home/yannvr/C4/django/db/models/fields/related.py" in flatten_data
  700. instance_ids = [instance._get_pk_val() for instance in getattr(obj, self.name).all()]
File "/var/www/MT/django/db/models/query.py" in __iter__
  108. return iter(self._get_data())
File "/var/www/MT/django/db/models/query.py" in _get_data
  470. self._result_cache = list(self.iterator())
File "/var/www/MT/django/db/models/query.py" in iterator
  174. select, sql, params = self._get_sql_clause()
File "/var/www/MT/django/db/models/query.py" in _get_sql_clause
  484. joins2, where2, params2 = self._filters.get_sql(opts)
File "/var/www/MT/django/db/models/query.py" in get_sql
  648. joins2, where2, params2 = val.get_sql(opts)
File "/var/www/MT/django/db/models/query.py" in get_sql
  699. return parse_lookup(self.kwargs.items(), opts)
File "/var/www/MT/django/db/models/query.py" in parse_lookup
  833. joins2, where2, params2 = lookup_inner(path, lookup_type, value, opts, opts.db_table, None)
File "/var/www/MT/django/db/models/query.py" in lookup_inner
  940. raise TypeError, "Cannot resolve keyword '%s' into field" % name

The middleware running on my server are: MIDDLEWARE_CLASSES = (

'django.middleware.common.CommonMiddleware?', 'django.contrib.sessions.middleware.SessionMiddleware?', 'django.contrib.auth.middleware.AuthenticationMiddleware?', 'django.middleware.doc.XViewMiddleware',

)

The moment is goes wrong is when trying to lookup for that field

File "/var/www/MT/django/db/models/query.py" in __iter__
  108. return iter(self._get_data())

self= Error in formatting:Cannot resolve keyword 'mtreqroute' into field

Patch to fix this in the modpython handler.

Just got bit by this one again.

It only appeared in modpython, not in the shell, and not using the built-in server (runserver).

I've attached a patch that forces preemptive model loading in the modpython handler. As with the management shell patch, this should be considered a temporary workaround.

Patch for fastcgi

Workaround for stand-alone scripts, provided by Ben Slavin on django-users

i still get the error (with modpython.patch) in combination with TagField?. anybody know a fix for this?

it's still the same exception as i posted on 03/22/07 05:51:20

My application is rendering useless since the commenting system and user creation systems return this TypeError?.

In my scenario, it seems to trace back to anytime user.groups.all() is called. This is without extra patches on latest SVN using builtin server.

Should write a test for this, but i don't know how. Must able to reproduce this bug on the shell somehow. Any ideas?

Øyvind: It's typically been somewhat dependent on a number of things including hardware (different CPUs give different results; this I know from previous experience), OS, Python micro version and installed apps. A simple example would be nice, but it might also be difficult to construct.

One request for everybody posting here: We really don't need any more "it happens for me, too" comments. They adds no value and make it hard to separate the useful comments out. We know it happens. We even understand mostly why. I'll get back to looking at this pretty soon (within the week, most likely), since I'll have a block of time to focus on it. So if everybody can be patient for a little longer (and hold off on the "me, too!" posts, since it's sounding a bit too much like AOL here) we'll get it fixed as soon as we can.

Here's a simple example that returns the TypeError?. Also my python -v says: Python 2.4.3 (#1, Oct 18 2006, 15:01:36) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin

I also asked axiak on IRC to try it and the test passed for him. He's on x86_64 though.

Thanks for all the hard work, hope this helps.

from django.contrib.auth.models import User,Group import datetime, random, sha import unittest

class ProfileTest?(unittest.TestCase?):

""" Used as a test case for what appears as this bug : http://code.djangoproject.com/ticket/1796 """

def setUp(self):

self.salt = sha.new(str(random.random())).hexdigest()[:5]

def testAcctCreate(self):

self.u=User(

first_name='bob', last_name='jones', username='bjones', email='test@bjones.com', # is_staff=True,

) self.u.set_password('123!@#') self.u.save() self.group=Group(name='Test Group') self.group.save() self.u.groups.add(self.group) self.assertEquals(User.objects.get(username='bjones'), self.u) self.assertEquals(Group.objects.get(name='Test Group'), self.group) print self.u.dict print self.u.groups.all()

Fixing the wiki formatting from the last ticket so that it's readable:

from django.contrib.auth.models import User,Group import datetime, random, sha import unittest

class ProfileTest?(unittest.TestCase?):

    """ Used as a test case for what appears as this bug : http://code.djangoproject.com/ticket/1796 """

    def setUp(self):

        self.salt = sha.new(str(random.random())).hexdigest()[:5]

    def testAcctCreate(self):

        self.u=User(

            first_name='bob', last_name='jones', username='bjones', email='test@bjones.com', # is_staff=True,

        )
        self.u.set_password('123!@#')
        self.u.save()
        self.group=Group(name='Test Group') 
        self.group.save()
        self.u.groups.add(self.group)
        self.assertEquals(User.objects.get(username='bjones'), self.u)   
        self.assertEquals(Group.objects.get(name='Test Group'), self.group)
        print self.u.dict print self.u.groups.all()

Sorry about that, I'm new to this tracker.

http://dpaste.com/hold/11109/

Also my python -v says:

Python 2.4.3 (#1, Oct 18 2006, 15:01:36) 
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin

Mr. Anonymous, please don't remove the flag "has patch" without giving a reason.

This ticket obviously has a lot of patches, but they need to be combined --> needs improved patch.

I'm removing the "has patch" notation (it wasn't me above, but this reminded me), since none of the attached patches are fixes for this ticket and aren't going to get checked in. Even a combination of them isn't a fix, because they are workarounds, rather than fixing the root cause.

Leaving the patches attached so that people have workarounds available until this is fixed properly, though.

The others didn't work for me. This is another workaround for standalone scripts, with a different approach. Works in VERSION (0,95.1, None)

I'm not sure where everyone is on this but I figured it out for me so I thought I'd share:

The problem is that the get_all_related_many_to_many_objects() caches itself, so if the many-to-many manager gets called before all the apps are loaded, any models in apps added afterwards won't be in that cache. I commented out the line that gets the cache so it has to re-cache each time, and it fixes the problem. Behold:

def get_all_related_many_to_many_objects(self):     
        try: # Try the cache first.    
            return self.ham_sandwich
            #return self._all_related_many_to_many_objects
        except AttributeError:    
            ...

Since .ham_sandwich is never found it falls into the AttributeError? exception and re-caches. It then works as planned.

The permanent fix, it seems to me, is to make it so that when a many-to-many is created, it clears the _all_related_many_to_many_objects of the _meta of the relation class, but I'm not sure exactly where best to do that.

Oh and as a temporary fix for everyone bitten by the bug, you can delete the cache after the creation of the model that isn't able to get the relation. Like so:

from other_app.models import Tag

class Thing(models.Model):
    tag = models.ManyToManyField(Tag)

del Tag._meta._all_related_many_to_many_objects

In case it's helpful in further debugging: I've got an app which pretty reliably triggers this if I hit the public detail view of an object immediately after starting/restarting the server (Apache/mod_python). If, on the other hand, I hit the admin change page for the same object first, everything's fine. I assume this is because the admin's doing hardcore stuff to find all the installed models.

Replying to brantley <deadwisdom@gmail.com>:

I'm not sure where everyone is on this but I figured it out for me so I thought I'd share: The problem is that the get_all_related_many_to_many_objects() caches itself, so if the many-to-many manager gets called before all the apps are loaded, any models in apps added afterwards won't be in that cache. I commented out the line that gets the cache so it has to re-cache each time, and it fixes the problem. Behold: {{{ def get_all_related_many_to_many_objects(self): try: # Try the cache first. return self.ham_sandwich #return self._all_related_many_to_many_objects except AttributeError?: ... }}} Since .ham_sandwich is never found it falls into the AttributeError? exception and re-caches. It then works as planned. The permanent fix, it seems to me, is to make it so that when a many-to-many is created, it clears the _all_related_many_to_many_objects of the _meta of the relation class, but I'm not sure exactly where best to do that.

An FYI...this suggested fix did not work for me.

(In [5919]) Rewrote portions of the app- and model-cache initialisation to handle some corner cases. It is now possible to use m2m relations before everything is imported and still get the right results later when importing is complete. Also, get_apps() should always return the same results, so apps won't randomly disappear in the admin interface.

Also reorganised the structure of loading.py, since the number of global variables was exploding. The public API is still backwards compatible.

Fixed #1796 and #2438 (he claims, optimistically).

u = User.objects.get(username='mandric') u.groups.all() now works!

woohoo!!

I can confirm that this [5919] has resolved the issue I was having with the admin interface.