Bigint field object needed

Additionally, a signedness boolean option is needed on IntegerField?.

See bigint_patch.txt for a BigIntegerField? and PositiveBigIntegerField?. I took it for a spin on mysql but it needs testing on postgres and sqlite (the backend-specific stuff is there but has not been tested).

jmadson: I think PositiveIntegerField?, SmallPositiveIntegerField?, or BigPostitiveIntegerField? might be what you're looking for unless I misunderstood.

BigIntegerField? seems OK on PostgreSQL, but PostitiveBigIntegerField? throws ProgrammingError: ERROR: integer out of range for 18446744073709551615. I'll see what I can do.

See my post for details, but it looks like a PostitiveBigInteger? on PostgreSQL is only going to reach 9 bajillion whereas MySQL does 18 bajillion. Still needs testing on sqlite.

Matt Croydon patch is now quite dated for the existing codebase. I have updated ticket #399 with the new patch. It is called bigint_patch_06aug06.txt. Since I am most familiar with MySQL, I modified the backend code for MySQL. The changes for Oracle, postgres, sqlite, etc. should be minimal, and should be done for django/db/backends. I have done some simple tests. Please take it out for a spin.

I have updated Gopal (previously Matt Croydon) patch against revision [4274]
See bigint_patch_03jan07.diff

Ok, on second thoughts this is a valid ticket.

Current patch needs some documentation and implementation for other database backends.

Last patch with docs added.

needs_docs = 0 and stripping redundant prefix from summary.

Add BigIntegerField? support to django (SQLite, MySQL, Postgresql, Oracle, MSSQL)

Add BigIntegerField?? support to django (SQLite, MySQL, Postgresql (and postgresql_psycopg2), Oracle, MSSQL)

Oops. I missed introspection support for postgresql_psycopg2. Updated patch attached.

The patch django-bigint-20070712.patch is working for me (psycopg2)

Previous patch with simple tests

Updated, added constraint checking

The latest patch's approach for constraint checking (sys.maxint) isn't going to work for 32bit systems.

sys.maxint was changed to class constant.

This isn't yet ready for checkin. The MAX_BIGINT variable in django.db.models.fields is fixed, but in reality is dependant on both architecture and database backend. I'm not sure what the fix is, but hard-coding 64 bits is going to look as silly in ten years as hardocoding 32 bits does today.

Replying to jacob:

This isn't yet ready for checkin. The MAX_BIGINT variable in django.db.models.fields is fixed, but in reality is dependant on both architecture and database backend. I'm not sure what the fix is, but hard-coding 64 bits is going to look as silly in ten years as hardcoding 32 bits does today.

But we _define_ BIGINT as 64 bits long (as written in doc). So why not to hardcode it that it will be always 64bit? Also in all db backends it is defined as 64bit type. So, there is no same situation like with integer data type which has always denpended on size of CPU register. This will always be 64bits. If there is a variant that db backend cannot save 64bit value, then we cannot us bigint type at all. But all backends supports 64bit values, so there is no problem with this. Yes, I also don't like `magic' value, but I think that it is adequate here because python doesn't define anything similar to sys.maxint.

Typo in latest patch:

raise ValueError("Value is so small/large to fit to field")

should be

raise ValueError("Value is too small/large to fit this field")

(or something like that. Also, same message is in tests/modeltests/field_types/models.py)

The SQL specification defines bigint as a 64-bit signed integer, and as such the definition of bigint in the context of SQL will never change, just as the definition of int has not changed despite the need for 64-bit signed integers today. Thus I believe jacob's reason for delaying check-in of this patch to be invalid. Please change check this in as soon as possible.

"Ready for checkin" is a determination made by triagers and committers. Please don't set it unless you're one of those people.

English typos

Replying to corporeal: Agreed, changing to 'ready for checkin'

Please, please read http://code.djangoproject.com/ticket/399#comment:24 and http://www.djangoproject.com/documentation/contributing/#ticket-triage

Replying to ramiro:

Please, please read http://code.djangoproject.com/ticket/399#comment:24 and http://www.djangoproject.com/documentation/contributing/#ticket-triage

Ticket triagers: community members who keep track of tickets, making sure the tickets are always categorized correctly.

Probably I don't understand this correctly. Who are triagers if not us? Is there any list of them? How to make some progress with tickets, whose are ready for months and no one moves them to next stage?

Recalling comments, to avoid triagers loosing time reading the full log to take conclusions:

• In comment:20 jacob says that why would we hardcode that Bigint field to 64bits. Thus marking as "needs improvement".
• But corporeal says: "The SQL specification defines bigint as a 64-bit signed integer"

So, hardcoding it is not that bad. maybe we could ask the backend about the bigint size, but writting this for every backend knowing that BIGINT **must be** 64bits long seems a bit cumbersome.

So then, I'm setting "needs improvement" to 0, any triager please move to "Ready for checkin" or "Please ask every backend if they feel right beeing 64bit long" :)

Sorry for meddling in, but shouldn't django/db/models/fields/init.py have

if value > self.MAX_BIGINT or value < -self.MAX_BIGINT - 1

instead of

if value > self.MAX_BIGINT or value < -self.MAX_BIGINT

?

Replying to honeyman:

Sorry for meddling in, but shouldn't django/db/models/fields/init.py have if value > self.MAX_BIGINT or value < -self.MAX_BIGINT - 1 instead of if value > self.MAX_BIGINT or value < -self.MAX_BIGINT ?

You're right. Appending new patch.

-1 problem fix

The patch I just submitted fixes an "integer out of range" in pgsql by specifying the internal type of the BigIntegerField?

There was a lot of work on this, and then it all just stopped, waiting for Triage team to move to check-in. I would like to repeat permon's question above.

How do we give a "shout-out" to the triage members when a ticket has met check-in requirements, but yet has been sitting for months, particularly when we don't know who they are? (I understand it's a volunteer effort and all, but it's still a valid question).

I really need this patch, and it seems like many others do as well. As I am working on the newforms-admin branch, I would like to not have to apply this patch everytime I update SVN.

Restored tests to previously submitted patch

Yep, this does look ready for checkin to me. The best way to get tickets rolling again is to ask about them in django-dev group. Would you mind doing that now?

I'll promote anyway.

Replying to SmileyChris:

Yep, this does look ready for checkin to me. The best way to get tickets rolling again is to ask about them in django-dev group. Would you mind doing that now?

Thank you. I will do so right away.

Hi, I have an issue with the latest patch (bigint-patch-2008-04-25.diff) and fixtures. When I load a fixtures file that contains BigInteger? fields, I have this error : "Value is too small/large to fit this field"

In file django/db/models/fields/init.py in class BigIntegerField? in function get_db_prep_save(self, value). The type of value parameter is unicode instead of long. (The content of value match correctly with the biginteger field in my fixtures file).

If I cast value parameter to long in the condition statement all works fine, here is my update :

    def get_db_prep_save(self, value):
        value_long =  long(value)
        if value_long > self.MAX_BIGINT or value_long < -self.MAX_BIGINT - 1: 
            raise ValueError("Value is to small/large to fit this field") 
        return super(BigIntegerField, self).get_db_prep_save(value)

The value should have already been converted to a Long or an Integer by that point in the code. So the problem lies elsewhere. JSON and YAML serialization work just fine. It's only XML that is producing this error. From what I can tell, a regular IntegerField? should also be producing this error, but for some reason it is not, and I cannot determine why not.

In any case, the proper solution for something like this is to override to_python():

    def to_python(self, value):
        return super(BigIntegerField, self).to_python(long(value))

I'm changing this bug to Unreviewed, as it's obviously not ready for checkin.

Ok. I got it. The problem is introduced by trying to perform validation, and since IntegerField? does no validation at all - by design - it produces no error here. Model is smart enough to parse out the integer by the time it gets to the DB.

But there is a bigger question: Why are we validating this field at all? Why not rely upon the backend to provide its own validation, just like we do with IntegerField??

Seems to me the fix is to skip validation entirely. That would also avoid the whole mess with different bigint sizes on different architectures.

For anyone who would like BigInteger? support, there is no need to wait for or to apply this patch. If you are using newforms, the following is all you need:

from django.db.models.fields import IntegerField
from django.conf import settings

class BigIntegerField(IntegerField):
	empty_strings_allowed=False
	def get_internal_type(self):
		return "BigIntegerField"
	
	def db_type(self):
		return 'NUMBER(19)' if settings.DATABASE_ENGINE == 'oracle' else 'bigint'

Replying to mwdiers <martin@diers.us>:

Ok. I got it. The problem is introduced by trying to perform validation, and since IntegerField? does no validation at all - by design - it produces no error here. Model is smart enough to parse out the integer by the time it gets to the DB. But there is a bigger question: Why are we validating this field at all? Why not rely upon the backend to provide its own validation, just like we do with IntegerField?? Seems to me the fix is to skip validation entirely. That would also avoid the whole mess with different bigint sizes on different architectures.

a) You're right, there is a possibility, that value arrives as string. So cast to long is necessary. b) Why there is validation -> main reason was that each backend behaves differently. If I remember well, SQLite saves clipped value, MySQL raises exception and so on. For me looked like best way to have same behaviour with all backends.

missing typecast

the current patch doesn't handle None values - get_db_prep_save() should explicitly check for it:

def get_db_prep_save(self, value):
    if value is not None:
        value_long =  long(value)
        if value_long > self.MAX_BIGINT or value_long < -self.MAX_BIGINT - 1: 
            raise ValueError("Value is to small/large to fit this field") 
    return super(BigIntegerField, self).get_db_prep_save(value)

I cannot speak for the developers, but when this patch has come up on the development group, time and again they have said that it is not needed, because it is now so easy to create your own special field types, on the fly, and in the case of a BigInt?, it is trivial to do so. See the above example which I gave, dated 5/2/08.

I completely agree with this, especially considering the direction that Python 3 seems to be heading, where there is no longer any distinction at all between an Int and a Long, and the fact that 64-bit architectures are quickly relegating the old Int/Long distinction to an historical observation.

I would not be surprised if they eventually close this ticket as a "wontfix".

Replying to mwdiers: There are two possibilities - we want to have BIGINT in all db backends, then this patch should be finished and incorporated into trunk. Second possibility is that bigint is not 'basic' field and we will discard this ticket. But it depends on somone from core developers. Again I think, that situation has changed after 1.0, so I will change status to 'design decision'.

Depending on result of this decision, I will try to finish the patch or discard it. From my point of view are both solutions valid.