Discussion:
Making max_length argument optional
Podrigal, Aron
2015-07-29 04:27:52 UTC
Permalink
Hi,

I am using postgresql and I prefer my VARCHAR columns not to have a length
limit. Is there any particular reason why max_length arg to fields is
required. If for compatibility with other database backends we can have
some sane default if it is None.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yjwzpk%2BN2%3D4WWbmj5zu2Gwu67G0Sme1f8xQa%2BKrzSQ2Rw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loïc Bistuer
2015-07-29 05:40:50 UTC
Permalink
Hi Aron,

I'm +1 on addressing this, I often don't care about max_length, but I still want a terse form representation. In my projects I use a subclass of TextField that sets a TextInput wiget in its formfield() method. But that's not very elegant: it requires a custom field for a common use-case, and it's tightly coupled to Django forms (i.e. won't work with other form libraries and non-HTML forms).

Now how we address this depends on the meaning we want to give to models fields like CharField or TextField, more specifically whether the emphasis is on end user's perspective or on mapping 1:1 to database concepts.

If we see CharField and TextField as respectively "short/single line field" and "long/multiline field" from a user's perspective, then it makes sense to lift the max_length requirement and use `db_type='TextField'` when max_length is None.

However if we want to stay close to their database equivalents, then we could have a display hint on TextField (e.g. TextField(display_multiline=False)) and Field.formfield() would use that hint to provide a TextInput widget instead of a Textarea.

Personally I'd much prefer option 1, it's significantly less verbose and more often than not I don't want a length constraint, especially considering that there is no performance penalty in doing so with all the databases that I care about. Also considering we want first class virtual field support (and plan on turning existing concrete fields into virtual fields) I don't think it's a big problem to break from the existing models.Field/db types mapping.

Cheers,
Loïc
Hi,
I am using postgresql and I prefer my VARCHAR columns not to have a length limit. Is there any particular reason why max_length arg to fields is required. If for compatibility with other database backends we can have some sane default if it is None.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yjwzpk%2BN2%3D4WWbmj5zu2Gwu67G0Sme1f8xQa%2BKrzSQ2Rw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/EB67D8D3-1633-4CE7-9A7D-E22FA1EB4A47%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Tim Graham
2015-07-29 13:28:50 UTC
Permalink
Ticket is https://code.djangoproject.com/ticket/14094
Post by Loïc Bistuer
Hi Aron,
I'm +1 on addressing this, I often don't care about max_length, but I
still want a terse form representation. In my projects I use a subclass of
TextField that sets a TextInput wiget in its formfield() method. But that's
not very elegant: it requires a custom field for a common use-case, and
it's tightly coupled to Django forms (i.e. won't work with other form
libraries and non-HTML forms).
Now how we address this depends on the meaning we want to give to models
fields like CharField or TextField, more specifically whether the emphasis
is on end user's perspective or on mapping 1:1 to database concepts.
If we see CharField and TextField as respectively "short/single line
field" and "long/multiline field" from a user's perspective, then it makes
sense to lift the max_length requirement and use `db_type='TextField'` when
max_length is None.
However if we want to stay close to their database equivalents, then we
could have a display hint on TextField (e.g.
TextField(display_multiline=False)) and Field.formfield() would use that
hint to provide a TextInput widget instead of a Textarea.
Personally I'd much prefer option 1, it's significantly less verbose and
more often than not I don't want a length constraint, especially
considering that there is no performance penalty in doing so with all the
databases that I care about. Also considering we want first class virtual
field support (and plan on turning existing concrete fields into virtual
fields) I don't think it's a big problem to break from the existing
models.Field/db types mapping.
Cheers,
Loïc
Post by Podrigal, Aron
Hi,
I am using postgresql and I prefer my VARCHAR columns not to have a
length limit. Is there any particular reason why max_length arg to fields
is required. If for compatibility with other database backends we can have
some sane default if it is None.
Post by Podrigal, Aron
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
Post by Podrigal, Aron
To unsubscribe from this group and stop receiving emails from it, send
<javascript:>.
Post by Podrigal, Aron
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CANJp-yjwzpk%2BN2%3D4WWbmj5zu2Gwu67G0Sme1f8xQa%2BKrzSQ2Rw%40mail.gmail.com.
Post by Podrigal, Aron
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b6d01012-93bb-41b6-9065-c1d360e63b61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-07-29 16:25:26 UTC
Permalink
I see models as what defines the database ddl and and not a representation
from a end users perspective. And I see the tight coupling between the 2
improper. Although in most cases the assumptions of representation and
mapping of the 2 is mostly accepted, explicit is better than implicit. Of
course we can have defaults that would work for most cases, but we need a
way to override that. For me I rarely use the builtin generated widgets, I
use restframework so all I'm interested in a model is for db definition in
a highly customizable way.

I would opt for a db_length kwarg which can in turn be used for any field
type eg. Timestamps
Post by Tim Graham
Ticket is https://code.djangoproject.com/ticket/14094
Post by Loïc Bistuer
Hi Aron,
I'm +1 on addressing this, I often don't care about max_length, but I
still want a terse form representation. In my projects I use a subclass of
TextField that sets a TextInput wiget in its formfield() method. But that's
not very elegant: it requires a custom field for a common use-case, and
it's tightly coupled to Django forms (i.e. won't work with other form
libraries and non-HTML forms).
Now how we address this depends on the meaning we want to give to models
fields like CharField or TextField, more specifically whether the emphasis
is on end user's perspective or on mapping 1:1 to database concepts.
If we see CharField and TextField as respectively "short/single line
field" and "long/multiline field" from a user's perspective, then it makes
sense to lift the max_length requirement and use `db_type='TextField'` when
max_length is None.
However if we want to stay close to their database equivalents, then we
could have a display hint on TextField (e.g.
TextField(display_multiline=False)) and Field.formfield() would use that
hint to provide a TextInput widget instead of a Textarea.
Personally I'd much prefer option 1, it's significantly less verbose and
more often than not I don't want a length constraint, especially
considering that there is no performance penalty in doing so with all the
databases that I care about. Also considering we want first class virtual
field support (and plan on turning existing concrete fields into virtual
fields) I don't think it's a big problem to break from the existing
models.Field/db types mapping.
Cheers,
Loïc
Post by Podrigal, Aron
Hi,
I am using postgresql and I prefer my VARCHAR columns not to have a
length limit. Is there any particular reason why max_length arg to fields
is required. If for compatibility with other database backends we can have
some sane default if it is None.
Post by Podrigal, Aron
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
Post by Podrigal, Aron
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CANJp-yjwzpk%2BN2%3D4WWbmj5zu2Gwu67G0Sme1f8xQa%2BKrzSQ2Rw%40mail.gmail.com.
Post by Podrigal, Aron
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/b6d01012-93bb-41b6-9065-c1d360e63b61%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/b6d01012-93bb-41b6-9065-c1d360e63b61%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110' '1010000', '1101111',
'1100100', '1110010', '1101001', '1100111', '1100001', '1101100'

P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yiFa0FeSc3of21MoKM_sFrfGkrk-6sHH11yXQDqB7Ldcg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Aymeric Augustin
2015-07-30 09:15:32 UTC
Permalink
I see models as what defines the database ddl and and not a representation from a end users perspective.
Django models do a bit more than this: see the `validators` argument, EmailField, etc.
And I see the tight coupling between the 2 improper.
I understand your perspective; it's a trade-off between purity and convenience and the optimal answer depends on each project.
For me I rarely use the builtin generated widgets, I use restframework so all I'm interested in a model is for db definition in a highly customizable way.
Please make sure your proposals also account for the various way Django can be and has been used :-)
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2A451482-2DDD-42CE-895B-55E1FF1026D3%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-09-21 04:15:00 UTC
Permalink
Iv'e started to work on this ticket [1] today and here [2] is my work so
far. I'd appreciate some feedback on the implementation.
There are a couple ways to handle the database type length.

1) Separate types one with a length spec and one without (as mentioned
by Ben Davis in the ticket)
2) Some interface to get the type length specs (the approach I took)

The second option more generic, and provides an interface for any type to
specify a length or scale precision.

[1] https://code.djangoproject.com/ticket/14094
[2]
https://github.com/django/django/compare/master...ar45:max_length_optional

On Thu, Jul 30, 2015 at 5:15 AM, Aymeric Augustin <
Post by Podrigal, Aron
I see models as what defines the database ddl and and not a
representation from a end users perspective.
Django models do a bit more than this: see the `validators` argument, EmailField, etc.
Post by Podrigal, Aron
And I see the tight coupling between the 2 improper.
I understand your perspective; it's a trade-off between purity and
convenience and the optimal answer depends on each project.
Post by Podrigal, Aron
For me I rarely use the builtin generated widgets, I use restframework
so all I'm interested in a model is for db definition in a highly
customizable way.
Please make sure your proposals also account for the various way Django
can be and has been used :-)
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/2A451482-2DDD-42CE-895B-55E1FF1026D3%40polytechnique.org
.
For more options, visit https://groups.google.com/d/optout.
--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110' '1010000', '1101111',
'1100100', '1110010', '1101001', '1100111', '1100001', '1101100'

P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yhpwRu2iwRU5TDf3NGbvVAYEnJPorjn3tAON0CHuTPEOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Paulo Maciel
2015-09-21 16:21:56 UTC
Permalink
+1 max_length optional
Post by Podrigal, Aron
Hi,
I am using postgresql and I prefer my VARCHAR columns not to have a length
limit. Is there any particular reason why max_length arg to fields is
required. If for compatibility with other database backends we can have
some sane default if it is None.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b0cfde2d-195d-487b-90de-55219030f898%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Tom Evans' via Django developers (Contributions to Django itself)
2015-09-21 16:54:44 UTC
Permalink
On Mon, Sep 21, 2015 at 5:21 PM, Paulo Maciel
Post by Paulo Maciel
+1 max_length optional
I'm slightly worried from a DB point of view. Whilst blessed
PostgreSQL supports variable length fields seemingly without issues,
other DB engines (MySQL, for instance), have significant performance
issues using a TEXT field versus a VARCHAR field - particularly on
indexed fields, it must do a row read to retrieve data, rather than
use an indexed value.

Remember that VARCHAR columns can be fully indexed, whilst TEXT
columns can only have the first N (specified by user) characters
indexed, and reading the data will always require a row read, even if
the text is shorter than the indexed size.

In the ticket, it states that

"""
If I read the oracle docs correctly, specifying a length is mandatory
for VARCHAR2, and the maximum is 4000 bytes. For MySQL, it's 65K
bytes, for PostgreSQL and SQLite it's very large.
"""

For MySQL, it is 64k (65,535 bytes is *not* 65k!), and that is not the
whole story - 64k is the maximum record size - all your columns must
fit within that limit, not just one column in the record.

http://dev.mysql.com/doc/refman/5.7/en/column-count-limit.html

I'm slightly concerned as a MySQL/MariaDB user that this will lead to
"reusable" apps that are not performant on MySQL. Given that we
(generally) let Django define the database tables based upon models,
it seems much more important that the definition is correct and high
performing in all supported backends than to avoid choosing what
database columns you require in the tables - after all, that is what
the TextField is for, to allow the programmer to specify "this is an
arbitrary length text field".

I'm all for DB engines that support arbitrary length varchar fields to
use that when you specify a TextField. I'm very not keen on having
MySQL represent the majority of char fields as TEXT because no-one
could be bothered to specify the appropriate length.

Cheers

Tom
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFHbX1KnWTfw6Hk18Buu3hZOiHVB2RW4JNk_PFdaO_QoLK6CmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Christophe Pettus
2015-09-21 18:46:32 UTC
Permalink
Post by 'Tom Evans' via Django developers (Contributions to Django itself)
I'm slightly worried from a DB point of view.
I have to agree, even speaking as PostgreSQL geek. While VARCHAR and TEXT are implemented the same way in PostgreSQL, conceptually they're different things. I don't think the relatively modest benefit of having no default justifies the problems that result on other platforms.

--
-- Christophe Pettus
***@thebuild.com
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/DA725AF8-A7CA-449F-B92A-0BCCDB124AD6%40thebuild.com.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-09-21 21:49:32 UTC
Permalink
We're not talking about representing all CharFields as TEXT, it is about
choosing a sane length as the default for the varchar datatype. So if you
would not specify max_length, for MySQL it would be varchar(255), for
oracle it would be varchar(4000 / bytes_per_char_for NLS_CHARACTERSET) and
for PostgreSQL it would be just VARCHAR without a length.

I would like some comments / help for how to compute the size of the string
in bytes for Oracle. unless we can opt for using a default length of 255
for all database backends besides postgres.
On Sep 21, 2015, at 9:54 AM, 'Tom Evans' via Django developers
Post by 'Tom Evans' via Django developers (Contributions to Django itself)
I'm slightly worried from a DB point of view.
I have to agree, even speaking as PostgreSQL geek. While VARCHAR and TEXT
are implemented the same way in PostgreSQL, conceptually they're different
things. I don't think the relatively modest benefit of having no default
justifies the problems that result on other platforms.
--
-- Christophe Pettus
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/DA725AF8-A7CA-449F-B92A-0BCCDB124AD6%40thebuild.com
.
For more options, visit https://groups.google.com/d/optout.
--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110' '1010000', '1101111',
'1100100', '1110010', '1101001', '1100111', '1100001', '1101100'

P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yiKv1iop5nJ%2BkH444L4iJKCATHt%2Bdkd%3DFiC872mgYUVmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Christophe Pettus
2015-09-22 00:42:28 UTC
Permalink
We're not talking about representing all CharFields as TEXT, it is about choosing a sane length as the default for the varchar datatype.
But that means notably different schemas on different backends, for not an obvious gain. What's the benefit there?

--
-- Christophe Pettus
***@thebuild.com
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/FE33D41D-30F7-497F-9E2D-4ABC396E4BE6%40thebuild.com.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-09-22 00:49:02 UTC
Permalink
Different schemas?? Schema will always be different for each database
backend according to its datatypes. I really don't understand what your
concern is. In any case your free to specify a max_length=N where it will
be the same for all backends.
Post by Podrigal, Aron
Post by Podrigal, Aron
We're not talking about representing all CharFields as TEXT, it is about
choosing a sane length as the default for the varchar datatype.
But that means notably different schemas on different backends, for not an
obvious gain. What's the benefit there?
--
-- Christophe Pettus
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/FE33D41D-30F7-497F-9E2D-4ABC396E4BE6%40thebuild.com
.
For more options, visit https://groups.google.com/d/optout.
--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110' '1010000', '1101111',
'1100100', '1110010', '1101001', '1100111', '1100001', '1101100'

P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yg%2Be6Oco8D74Fd1K4QWrZwv8itp5nNZc75y3uoDiKMmwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Christophe Pettus
2015-09-22 00:52:32 UTC
Permalink
Different schemas?? Schema will always be different for each database backend according to its datatypes.
It means if you specify a CharField without a length, you don't know how many characters it can accept without error. That doesn't seem like something we should make a change to accept.
I really don't understand what your concern is.
The current behavior seems entirely reasonable, and I'm not sure I understand what problems it is causing. Specifying a maximum length on a CharField is not just a random habit; it should be done as a way of sanity checking the value to a reasonable length. Sometimes, that's natural to the data (there are no 50 character telephone numbers or 5000 character email addresses), sometimes it's just a way of making sure that something bad doesn't get into the database.

--
-- Christophe Pettus
***@thebuild.com
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/15D5715A-3EF5-4CC0-830C-8EB714424335%40thebuild.com.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-09-22 01:12:15 UTC
Permalink
The reason for having a max_length set to None, is because that's what I
want for my database columns to be in Postgres, and for MySQL I don't care
about the length too, I always choose varchar(255) just for because it is
required for the database backend. And for validation of the max length, I
would argue that just like validating the max I want to validate the min.
In short, I want to have more control on my database schema.

And for your concern, there will be a MaxLengthValidator added to the
validators to validate the users input does not exceed the database
backends maximum length just like when you set max_length explicitly. Also
you can manually provide a validator if your user input validation rule is
to be less than the db maximum length. See [1]. The
MaxLengthValidator should actually only be added if there isn't yet
any MaxLengthValidator explicitly given.


[1]
https://github.com/django/django/blob/9a97f847109d377995a2312e3d569a3dad1da677/django/db/models/fields/__init__.py#L1102-L1109
Post by Podrigal, Aron
Post by Podrigal, Aron
Different schemas?? Schema will always be different for each database
backend according to its datatypes.
It means if you specify a CharField without a length, you don't know how
many characters it can accept without error. That doesn't seem like
something we should make a change to accept.
Post by Podrigal, Aron
I really don't understand what your concern is.
The current behavior seems entirely reasonable, and I'm not sure I
understand what problems it is causing. Specifying a maximum length on a
CharField is not just a random habit; it should be done as a way of sanity
checking the value to a reasonable length. Sometimes, that's natural to
the data (there are no 50 character telephone numbers or 5000 character
email addresses), sometimes it's just a way of making sure that something
bad doesn't get into the database.
--
-- Christophe Pettus
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/15D5715A-3EF5-4CC0-830C-8EB714424335%40thebuild.com
.
For more options, visit https://groups.google.com/d/optout.
--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110' '1010000', '1101111',
'1100100', '1110010', '1101001', '1100111', '1100001', '1101100'

P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yjm-wLsR4neSjM5q_-9d5LNOfDXUmnyAan1zB2emdR%3D_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Christophe Pettus
2015-09-22 02:03:07 UTC
Permalink
The reason for having a max_length set to None, is because that's what I want for my database columns to be in Postgres, and for MySQL I don't care about the length too, I always choose varchar(255) just for because it is required for the database backend.
Well, that's not a practice I think we need to go to great lengths to support. If you *really* *must* have a VARCHAR field without a length, you can always use a migration to strip it off.

--
-- Christophe Pettus
***@thebuild.com
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/95E76678-2FAB-46B4-B830-0AC877886EE4%40thebuild.com.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-09-22 02:10:26 UTC
Permalink
There is actually another reason to not have to specify a max_length which
was mentioned earlier, is because most of the time you don't care about
that and is just tedious to have to specify that when you can get it to
work without it. Default values has always been here for that reason.
Post by Podrigal, Aron
Post by Podrigal, Aron
The reason for having a max_length set to None, is because that's what I
want for my database columns to be in Postgres, and for MySQL I don't care
about the length too, I always choose varchar(255) just for because it is
required for the database backend.
Well, that's not a practice I think we need to go to great lengths to
support. If you *really* *must* have a VARCHAR field without a length, you
can always use a migration to strip it off.
--
-- Christophe Pettus
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/95E76678-2FAB-46B4-B830-0AC877886EE4%40thebuild.com
.
For more options, visit https://groups.google.com/d/optout.
--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110' '1010000', '1101111',
'1100100', '1110010', '1101001', '1100111', '1100001', '1101100'

P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yi2mc4bVompW8kL_QYdr80kWEr42r7sj9pOjnitc%3DV0%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Christophe Pettus
2015-09-22 02:16:18 UTC
Permalink
There is actually another reason to not have to specify a max_length which was mentioned earlier, is because most of the time you don't care about that and is just tedious to have to specify that when you can get it to work without it. Default values has always been here for that reason.
I'm afraid I must disagree that "most of the time you don't care about it." I certainly do. I'm always reasonably careful to specify a max_length that corresponds to the underlying data.

There's no "sensible" default for max_length. It entirely depends on the data you are storing. Picking a backend-specific max_length means that application writers now have no idea how much data the CharField can store: 1GB-ish-depending-on-encoding? 255? 4000 or something less depending on Oracle's encoding?

Requiring a max_length is enforcing a good practice.
--
-- Christophe Pettus
***@thebuild.com
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/46F57CA5-459B-4B2D-B610-EF3FF1190256%40thebuild.com.
For more options, visit https://groups.google.com/d/optout.
Shai Berger
2015-09-22 02:22:10 UTC
Permalink
Hi,

I just went over the ticket and thread. There seem to be two concerns mixed
here.

Most people who expressed support for the feature have just been annoyed at
the need to specify a length when all they want to say is "a reasonably short
string" or "a one-liner".

Aron, on the other hand, seems to aim mostly at an option to remove the length
limitation completely.

I can see why "unlimited" makes sense, sometimes, on Postgres. But I think
it's a bad default for general use, especially in reusable apps.

I also don't like the interpretation of "unlimited" as "backend-specified max",
basically for the reasons Christoph mentioned (same schema on different
backends), and also because I'm concerned about the performance implications
of unnecessarily large fields and about the ambiguity of introspection.

I'd solve the "need to specify" issue by setting a default that is
intentionally smaller than the smallest (core) backend limitation, say 128.
I"d make an "unlimited length text field" a new type of field, explicitly not
supported on MySql and Oracle; and I'd suggest that it can live outside core
for a while. so we may get an impression of how popular it really is.

My 2 cents,
Shai.
Christophe Pettus
2015-09-22 02:26:25 UTC
Permalink
Post by Shai Berger
I'd solve the "need to specify" issue by setting a default that is
intentionally smaller than the smallest (core) backend limitation, say 128.
I'd be OK with that. Not wild, because I think that having to specify max_length is good discipline, but not everyone likes oatmeal, either. :)
Post by Shai Berger
I"d make an "unlimited length text field" a new type of field, explicitly not
supported on MySql and Oracle; and I'd suggest that it can live outside core
for a while. so we may get an impression of how popular it really is.
We kind of have that: TextField. The problem is that TextField has very different performance characteristics and implementation details on PostgreSQL vs MySQL and Oracle. I don't think we need another: If you know you are running on PostgreSQL, you just use TextField, and if you are either targeting a different database, or writing one that runs on multiple ones, you probably want CharField with a specific length.

--
-- Christophe Pettus
***@thebuild.com
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3542EAF7-C2EB-4A4B-94F4-8C7A9EC4AC4E%40thebuild.com.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-09-22 02:33:35 UTC
Permalink
Post by Christophe Pettus
Post by Shai Berger
I'd solve the "need to specify" issue by setting a default that is
intentionally smaller than the smallest (core) backend limitation, say
128.
I'd be OK with that. Not wild, because I think that having to specify
max_length is good discipline, but not everyone likes oatmeal, either. :)
OK, this satisfies me too.
Post by Christophe Pettus
Post by Shai Berger
I"d make an "unlimited length text field" a new type of field,
explicitly not
Post by Shai Berger
supported on MySql and Oracle; and I'd suggest that it can live outside
core
Post by Shai Berger
for a while. so we may get an impression of how popular it really is.
We kind of have that: TextField. The problem is that TextField has very
different performance characteristics and implementation details on
PostgreSQL vs MySQL and Oracle. I don't think we need another: If you know
you are running on PostgreSQL, you just use TextField, and if you are
either targeting a different database, or writing one that runs on multiple
ones, you probably want CharField with a specific length.
While this makes sense, the use of TextField is also for the purpose of
having rendered a text widget for ModelForms. So we should allow a
max_length of None for both.
Post by Christophe Pettus
--
-- Christophe Pettus
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/3542EAF7-C2EB-4A4B-94F4-8C7A9EC4AC4E%40thebuild.com
.
For more options, visit https://groups.google.com/d/optout.
--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110' '1010000', '1101111',
'1100100', '1110010', '1101001', '1100111', '1100001', '1101100'

P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yj28u18uT8b0V5DunyfyC2fe5m1ftnTLrfEo8nE3U1UVg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Josh Smeaton
2015-09-22 04:22:33 UTC
Permalink
Post by Podrigal, Aron
While this makes sense, the use of TextField is also for the purpose of
having rendered a text widget for ModelForms. So we should allow a
max_length of None for both.

If you use Oracle (or mysql I guess), you get used to manually changing the
form field widget to be a TextArea if that's what you want for your UI.
Using TextField with Oracle is a horrible experience unless you must have
it.
Post by Podrigal, Aron
Post by Christophe Pettus
Post by Shai Berger
I'd solve the "need to specify" issue by setting a default that is
intentionally smaller than the smallest (core) backend limitation, say
128.
I'd be OK with that. Not wild, because I think that having to specify
max_length is good discipline, but not everyone likes oatmeal, either. :)
OK, this satisfies me too.
Post by Christophe Pettus
Post by Shai Berger
I"d make an "unlimited length text field" a new type of field,
explicitly not
Post by Shai Berger
supported on MySql and Oracle; and I'd suggest that it can live outside
core
Post by Shai Berger
for a while. so we may get an impression of how popular it really is.
We kind of have that: TextField. The problem is that TextField has very
different performance characteristics and implementation details on
PostgreSQL vs MySQL and Oracle. I don't think we need another: If you know
you are running on PostgreSQL, you just use TextField, and if you are
either targeting a different database, or writing one that runs on multiple
ones, you probably want CharField with a specific length.
While this makes sense, the use of TextField is also for the purpose of
having rendered a text widget for ModelForms. So we should allow a
max_length of None for both.
Post by Christophe Pettus
--
-- Christophe Pettus
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/3542EAF7-C2EB-4A4B-94F4-8C7A9EC4AC4E%40thebuild.com
.
For more options, visit https://groups.google.com/d/optout.
--
Aron Podrigal
-
'1000001', '1110010', '1101111', '1101110' '1010000', '1101111',
'1100100', '1110010', '1101001', '1100111', '1100001', '1101100'
P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/86d12c55-934a-4491-9429-583a1e2dcd12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Aymeric Augustin
2015-09-22 05:56:40 UTC
Permalink
Hi Shai,
Post by Shai Berger
I'd solve the "need to specify" issue by setting a default that is
intentionally smaller than the smallest (core) backend limitation, say 128.
I would pick the highest value supported by all core backends (probably 255
for MySQL, unless there’s something about indexes and multi-byte encodings
that I forget) in order to minimize the need to increase it.

If we go for a lower value, I suggest to pick something totally arbitrary like
100 to make it clear that it isn't a technical limitation.
Post by Shai Berger
I"d make an "unlimited length text field" a new type of field, explicitly not
supported on MySql and Oracle; and I'd suggest that it can live outside core
for a while. so we may get an impression of how popular it really is.
The main use case seems to be “VARCHAR() on PostgreSQL”. What about defining a
slight variant of CharField in django.contrib.postgres that merely makes the
max_length argument default to None?
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/17C37814-E6A8-4E27-B590-BF9FFF42CB20%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Collin Anderson
2015-09-22 12:40:38 UTC
Permalink
If anyone's curious, the mysql situation is as crazy as you might expect. :)

The max is only determined only by the total row size (65,535 bytes) and
the index size (767 bytes default).

Mysql defaults to only allowing 3-byte (no emoji) unicode characters, so
65,535/3=21,845 max across the entire row (the sum of all of the maxes of
all char/varchar columns), and each indexed field only gets 767/3=255
characters by default.

If you change to 4-byte unicode characters, (which you should, but django
doesn't really help you out #18392), your max_lengths can add up to
65,535/4=16,383 characters, and if you want the field to be indexed, you
only get 191 characters (using the default index size). It's possible to
only index the first 767/4=191 characters of the field, but django doesn't
really support that.

Basically, 255 works pretty well by default, allowing 65,535/3/255=85
3-byte fields per row, and indexes just work if you stick to the default
settings.

https://dev.mysql.com/doc/refman/5.0/en/char.html
https://code.djangoproject.com/ticket/18392


On Tue, Sep 22, 2015 at 1:56 AM, Aymeric Augustin <
Post by Aymeric Augustin
Hi Shai,
Post by Shai Berger
I'd solve the "need to specify" issue by setting a default that is
intentionally smaller than the smallest (core) backend limitation, say
128.
I would pick the highest value supported by all core backends (probably 255
for MySQL, unless there’s something about indexes and multi-byte encodings
that I forget) in order to minimize the need to increase it.
If we go for a lower value, I suggest to pick something totally arbitrary like
100 to make it clear that it isn't a technical limitation.
Post by Shai Berger
I"d make an "unlimited length text field" a new type of field,
explicitly not
Post by Shai Berger
supported on MySql and Oracle; and I'd suggest that it can live outside
core
Post by Shai Berger
for a while. so we may get an impression of how popular it really is.
The main use case seems to be “VARCHAR() on PostgreSQL”. What about
defining a
slight variant of CharField in django.contrib.postgres that merely makes the
max_length argument default to None?
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/17C37814-E6A8-4E27-B590-BF9FFF42CB20%40polytechnique.org
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFO84S5TUwLPf%3DRrGXL4%3Dvf3647CufOsijiznttVQF%2BxYHaBFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Michael Manfre
2015-09-22 14:00:27 UTC
Permalink
I'm -1 on having a default max_length on CharField. We should encourage
developers to be more explicit when defining their models, not less. I've
inherited way too many databases that suffered from "VARCHAR(255)" being
chosen as the arbitrary "any field storing even a few characters will be
this length".

Instead of adding a default (and encouraging bad database schemas), what
are thoughts about adding a helper function to allow people to define their
own new CharField as a one liner? E.g. ShortCharField =
make_max_length_char_field(255)

I agree that the Postgresql specific field should exist in
django.contrib.postgres.

Regards,
Michael Manfre
Post by Collin Anderson
If anyone's curious, the mysql situation is as crazy as you might expect. :)
The max is only determined only by the total row size (65,535 bytes) and
the index size (767 bytes default).
Mysql defaults to only allowing 3-byte (no emoji) unicode characters, so
65,535/3=21,845 max across the entire row (the sum of all of the maxes of
all char/varchar columns), and each indexed field only gets 767/3=255
characters by default.
If you change to 4-byte unicode characters, (which you should, but django
doesn't really help you out #18392), your max_lengths can add up to
65,535/4=16,383 characters, and if you want the field to be indexed, you
only get 191 characters (using the default index size). It's possible to
only index the first 767/4=191 characters of the field, but django doesn't
really support that.
Basically, 255 works pretty well by default, allowing 65,535/3/255=85
3-byte fields per row, and indexes just work if you stick to the default
settings.
https://dev.mysql.com/doc/refman/5.0/en/char.html
https://code.djangoproject.com/ticket/18392
On Tue, Sep 22, 2015 at 1:56 AM, Aymeric Augustin <
Post by Aymeric Augustin
Hi Shai,
Post by Shai Berger
I'd solve the "need to specify" issue by setting a default that is
intentionally smaller than the smallest (core) backend limitation, say
128.
I would pick the highest value supported by all core backends (probably 255
for MySQL, unless there’s something about indexes and multi-byte encodings
that I forget) in order to minimize the need to increase it.
If we go for a lower value, I suggest to pick something totally arbitrary like
100 to make it clear that it isn't a technical limitation.
Post by Shai Berger
I"d make an "unlimited length text field" a new type of field,
explicitly not
Post by Shai Berger
supported on MySql and Oracle; and I'd suggest that it can live outside
core
Post by Shai Berger
for a while. so we may get an impression of how popular it really is.
The main use case seems to be “VARCHAR() on PostgreSQL”. What about
defining a
slight variant of CharField in django.contrib.postgres that merely makes the
max_length argument default to None?
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/17C37814-E6A8-4E27-B590-BF9FFF42CB20%40polytechnique.org
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAFO84S5TUwLPf%3DRrGXL4%3Dvf3647CufOsijiznttVQF%2BxYHaBFg%40mail.gmail.com
<https://groups.google.com/d/msgid/django-developers/CAFO84S5TUwLPf%3DRrGXL4%3Dvf3647CufOsijiznttVQF%2BxYHaBFg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
GPG Fingerprint: 74DE D158 BAD0 EDF8
keybase.io/manfre
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGdCwBvhkYqb3OgfKqXHJGNyTZbTsqYcnV%2BfP0YdKn4W%3D%2B1_MA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Aymeric Augustin
2015-09-22 05:49:23 UTC
Permalink
Hi Aron,
And for your concern, there will be a MaxLengthValidator added to the validators to validate the users input does not exceed the database backends maximum length just like when you set max_length explicitly.
This isn’t possible in a project that uses multiple databases, say PostgreSQL and Oracle. The form layer cannot say which length is correct because it has doesn’t know in what database the data will be saved.
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2FF1A3B1-0167-4E52-9F99-EB5BDD6B81C5%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-09-22 05:57:53 UTC
Permalink
Aymeric, thanks for clarification, my bad I missed that one. We will be
using max_length=255 or 128 for the default as Shia proposed.

BTW how is this handled by integer_ranges on per database backend? Also,
using checks_framework can we validate this setting?
On Sep 22, 2015 1:49 AM, "Aymeric Augustin" <
Post by Loïc Bistuer
Hi Aron,
Post by Podrigal, Aron
And for your concern, there will be a MaxLengthValidator added to the
validators to validate the users input does not exceed the database
backends maximum length just like when you set max_length explicitly.
This isn’t possible in a project that uses multiple databases, say
PostgreSQL and Oracle. The form layer cannot say which length is correct
because it has doesn’t know in what database the data will be saved.
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/2FF1A3B1-0167-4E52-9F99-EB5BDD6B81C5%40polytechnique.org
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yi_6TK5bB-rd5d7XZJAwyKRPQthS%2BM1C0vZyqVK%2BY1d9w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Aymeric Augustin
2015-09-22 06:03:37 UTC
Permalink
We will be using max_length=255 or 128 for the default as Shia proposed.
Would you mind giving a few hours for other contributors to react? I was asleep while you were having this discussion; not every contributor is hooked to django-developers, especially technical board members.
BTW how is this handled by integer_ranges on per database backend?
I was asking myself the same question after seeing https://code.djangoproject.com/ticket/14094#comment:15 <https://code.djangoproject.com/ticket/14094#comment:15>. I don’t know; perhaps it doesn’t! It’s worth investigating.
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ECA9C917-6FB8-467B-97DC-9E4FE970BC9B%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Podrigal, Aron
2015-09-22 06:56:35 UTC
Permalink
Ok, I'll wait for other contributors to react.
On Sep 22, 2015 2:03 AM, "Aymeric Augustin" <
We will be using max_length=255 or 128 for the default as Shia proposed.
Would you mind giving a few hours for other contributors to react? I was
asleep while you were having this discussion; not every contributor is
hooked to django-developers, especially technical board members.
BTW how is this handled by integer_ranges on per database backend?
I was asking myself the same question after seeing
https://code.djangoproject.com/ticket/14094#comment:15. I don’t know;
perhaps it doesn’t! It’s worth investigating.
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/ECA9C917-6FB8-467B-97DC-9E4FE970BC9B%40polytechnique.org
<https://groups.google.com/d/msgid/django-developers/ECA9C917-6FB8-467B-97DC-9E4FE970BC9B%40polytechnique.org?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANJp-yhdepHYgQ9eUCWwFEtVBsDjA8JbAroOTBGM_ssAxWGhUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Tim Chase
2015-09-22 11:58:43 UTC
Permalink
Post by Aymeric Augustin
Post by Podrigal, Aron
And for your concern, there will be a MaxLengthValidator added to
the validators to validate the users input does not exceed the
database backends maximum length just like when you set
max_length explicitly.
This isn’t possible in a project that uses multiple databases, say
PostgreSQL and Oracle. The form layer cannot say which length is
correct because it has doesn’t know in what database the data will
be saved.
Could this be resolved with allowing max_length=None (or some other
atom such as just object() that would have a unique id() that could
be tested with "is") to specify that the back-end uses the max-allowed
value? For Postgres, this would then use VARCHAR() with no limit,
while on MySQL, that could be 255 or whatever.

One could then include a helper function that would use the current
settings to check a string (such as from a form) to ensure that it
doesn't exceed the field's database-specific max-length (or whatever
the specified max-length value is).

-tim
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20150922065843.37699cb6%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.
'Tom Evans' via Django developers (Contributions to Django itself)
2015-09-22 14:05:31 UTC
Permalink
On Tue, Sep 22, 2015 at 1:49 AM, Podrigal, Aron
Post by Podrigal, Aron
Different schemas?? Schema will always be different for each database
backend according to its datatypes. I really don't understand what your
concern is. In any case your free to specify a max_length=N where it will be
the same for all backends.
This change would allow a django site running on postgres that you
could not dump the data from and reimport in to mysql without data
loss. It would make it easy to write non-portable apps without
thinking.
Post by Podrigal, Aron
While this makes sense, the use of TextField is also for the purpose of
having rendered a text widget for ModelForms. So we should allow a
max_length of None for both.
(I'm replying to two emails from you in one, hope that is alright)

No, TextField is used to designate an arbitrary length text field.
Having a widget.Textarea as the default input for this field is just a
default, it is perfectly correct to override the defaults either for
CharField or TextField to provide the desired widget type.

One should not be using models.TextField because a widget.Textarea is
desired for forms, instead it is to tell the database backend that the
column for this field should be an arbitrary length text type.

If the problem that is to be solved is that there is no easy way to
specify an arbitrary length text field that uses an input, this can be
solved with a very simple class:

class TextFieldWithInput(models.TextField):
def formfield(self, **kwargs):
defaults = {'widget': forms.TextInput}
defaults.update(kwargs)
return super(TextfieldWithInput, self).formfield(**defaults)

The main argument seems to be "Its a pain to think about the size of
my data when defining my database tables"; if we aren't thinking about
it then, then when do we think about it? Its kind of important..

Cheers

Tom
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFHbX1LbVfR4LdaqfF2S266dxhYRKA5B_dvbBA17c720spJKeA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Remco Gerlich
2015-09-22 08:01:01 UTC
Permalink
Post by Podrigal, Aron
We're not talking about representing all CharFields as TEXT, it is about
choosing a sane length as the default for the varchar datatype. So if you
would not specify max_length, for MySQL it would be varchar(255), for
oracle it would be varchar(4000 / bytes_per_char_for NLS_CHARACTERSET) and
for PostgreSQL it would be just VARCHAR without a length.
I feel that IF a "sane length as the default" exists, then it must be the
same for all database backends. The errors you get from a ModelForm on a
web page shouldn't change depending on the database backend!

Maybe django.contrib.postgres could have a ArbitraryLengthCharField?

Greetings,
Remco Gerlich
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFAGLK1%2BEsapoVNLr7Cgz4YqY0ykTiNWD2zNLfC%2Bce446Pzuxw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Christophe Pettus
2015-09-22 17:13:45 UTC
Permalink
Post by Remco Gerlich
Maybe django.contrib.postgres could have a ArbitraryLengthCharField?
Just a note that, on PostgreSQL, that's exactly what TextField is. There might be a use for a field that creates a VARCHAR without length on PostgreSQL, but I can't think of it.

--
-- Christophe Pettus
***@thebuild.com
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/D6E9D391-883D-4CB7-B9E2-78D6ACB8F7A4%40thebuild.com.
For more options, visit https://groups.google.com/d/optout.
Tom Christie
2015-09-23 17:02:20 UTC
Permalink
I'm with Tom here.
Forcing `max_length` to always be set on CharField feels like the right
decision. Having a default there seems unnecessary obscure, and more likely
to lead to untested/unnoticed failure cases.
It *could* be that we'd allow `max_length=None` to explicitly turn off the
validation, but I'm not sure if that's better than forcing developers who
want unvalidated lengths to use TextField.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/dc97b2d7-870e-4952-b08e-428eaf250844%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Wim Feijen
2016-02-27 12:55:23 UTC
Permalink
Hi guys,

I'd like to reach consensus about
https://code.djangoproject.com/ticket/14094

In the discussion, I saw three options:

1. Adding a default max_length = None, which may in practice differ per
database backend.
2. Adding a default max_length of a value which can be used across
databases. Lengths which have been mentioned are: 100, 128, 191 and 255
characters and 4000 bytes.
3. Keep everything as is, being as explicit as possible.

Option 1 does not allow for apps and data to be reused across several apps
and several core developers do not support this idea.

I'm in favour of option 2, because it makes for shorter code and more
importantly, it helps programmers choose a value which actually can be
reused across multiple databases (I didn't know about the possible limit of
191 characters on mysql! Did you?).

I dislike how it is now (option 3), and I don't think it is beneficial here
to force people to be explicit, because people are now using arbitrary
values anyway. 255 is commonly used, the poll app of the Django tutorial
uses 200. Actually I believe it is a big mess in practice, I see defaults
of 100, 200, 255, 300, 1000 seemingly randomly used. For example, try
searching for CharField in your favourite apps or CMSes.

In addition, adding a default for max_length still allows to be explicit
when it matters. And it makes life simple when it doesn't matter, and
simple is better than complex.

What would be a good default then? I am in favour of making it as big as
possible while still working on all databases, which would either be 191 of
255 in order to support mysql.

Wim
Post by Tom Christie
I'm with Tom here.
Forcing `max_length` to always be set on CharField feels like the right
decision. Having a default there seems unnecessary obscure, and more likely
to lead to untested/unnoticed failure cases.
It *could* be that we'd allow `max_length=None` to explicitly turn off the
validation, but I'm not sure if that's better than forcing developers who
want unvalidated lengths to use TextField.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a2cfad14-4384-4118-bc28-d89099433ca5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Florian Apolloner
2016-02-27 13:01:51 UTC
Permalink
-1 on adding an arbitrary default like 100/128
 If people choose a
CharField the should set max_length manually. In that sense I am for option
3.

Cheers,
Florian
Post by Wim Feijen
Hi guys,
I'd like to reach consensus about
https://code.djangoproject.com/ticket/14094
1. Adding a default max_length = None, which may in practice differ per
database backend.
2. Adding a default max_length of a value which can be used across
databases. Lengths which have been mentioned are: 100, 128, 191 and 255
characters and 4000 bytes.
3. Keep everything as is, being as explicit as possible.
Option 1 does not allow for apps and data to be reused across several apps
and several core developers do not support this idea.
I'm in favour of option 2, because it makes for shorter code and more
importantly, it helps programmers choose a value which actually can be
reused across multiple databases (I didn't know about the possible limit of
191 characters on mysql! Did you?).
I dislike how it is now (option 3), and I don't think it is beneficial
here to force people to be explicit, because people are now using arbitrary
values anyway. 255 is commonly used, the poll app of the Django tutorial
uses 200. Actually I believe it is a big mess in practice, I see defaults
of 100, 200, 255, 300, 1000 seemingly randomly used. For example, try
searching for CharField in your favourite apps or CMSes.
In addition, adding a default for max_length still allows to be explicit
when it matters. And it makes life simple when it doesn't matter, and
simple is better than complex.
What would be a good default then? I am in favour of making it as big as
possible while still working on all databases, which would either be 191 of
255 in order to support mysql.
Wim
Post by Tom Christie
I'm with Tom here.
Forcing `max_length` to always be set on CharField feels like the right
decision. Having a default there seems unnecessary obscure, and more likely
to lead to untested/unnoticed failure cases.
It *could* be that we'd allow `max_length=None` to explicitly turn off
the validation, but I'm not sure if that's better than forcing developers
who want unvalidated lengths to use TextField.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/db2345be-ccf9-4305-be60-35c2a1434432%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Wim Feijen
2016-02-27 13:33:26 UTC
Permalink
Hi Florian,

Can you please elaborate why the current situation is better? In practice,
most people add max_length=255 everytime by hand. I value your opinion but
I fail to understand your reasoning. Could you please explain further (if
you have time)?

Wim
Post by Florian Apolloner
-1 on adding an arbitrary default like 100/128
 If people choose a
CharField the should set max_length manually. In that sense I am for option
3.
Cheers,
Florian
Post by Wim Feijen
Hi guys,
I'd like to reach consensus about
https://code.djangoproject.com/ticket/14094
1. Adding a default max_length = None, which may in practice differ per
database backend.
2. Adding a default max_length of a value which can be used across
databases. Lengths which have been mentioned are: 100, 128, 191 and 255
characters and 4000 bytes.
3. Keep everything as is, being as explicit as possible.
Option 1 does not allow for apps and data to be reused across several
apps and several core developers do not support this idea.
I'm in favour of option 2, because it makes for shorter code and more
importantly, it helps programmers choose a value which actually can be
reused across multiple databases (I didn't know about the possible limit of
191 characters on mysql! Did you?).
I dislike how it is now (option 3), and I don't think it is beneficial
here to force people to be explicit, because people are now using arbitrary
values anyway. 255 is commonly used, the poll app of the Django tutorial
uses 200. Actually I believe it is a big mess in practice, I see defaults
of 100, 200, 255, 300, 1000 seemingly randomly used. For example, try
searching for CharField in your favourite apps or CMSes.
In addition, adding a default for max_length still allows to be explicit
when it matters. And it makes life simple when it doesn't matter, and
simple is better than complex.
What would be a good default then? I am in favour of making it as big as
possible while still working on all databases, which would either be 191 of
255 in order to support mysql.
Wim
Post by Tom Christie
I'm with Tom here.
Forcing `max_length` to always be set on CharField feels like the right
decision. Having a default there seems unnecessary obscure, and more likely
to lead to untested/unnoticed failure cases.
It *could* be that we'd allow `max_length=None` to explicitly turn off
the validation, but I'm not sure if that's better than forcing developers
who want unvalidated lengths to use TextField.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/871a631d-7e0e-4f14-9f90-ec629d8de26f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Florian Apolloner
2016-02-27 14:56:25 UTC
Permalink
Hi Wim,
Post by Wim Feijen
Can you please elaborate why the current situation is better?
I wouldn't say it is better, just equally good/bad ;)
Post by Wim Feijen
In practice, most people add max_length=255 everytime by hand. I value
your opinion but I fail to understand your reasoning. Could you please
explain further (if you have time)?
In practice, that would mean most people do not care about the length, so
they should be using a TextField, no? (See next paragraph for even another
option) If you want something which works on multiple databases, 255 is a
nonsensical default anyways -- with all the love for emojis out there,
you'd be limiting yourself to 191 (thank MySQL for that 767/4 = 191). What
worries me even more, is that changing this "default" (to accommodate for a
new database for instance) will require database migrations, so all of a
sudden upgrading django will result in migrations for all of your apps.

That all said, if it were for me I'd deprecate CharField and only support
TextField and give it an optional max_length (and a way to switch between
TextArea and LineEdit for UI) -- but that is probably just because on
Postgres there is no fundamental difference between them. From an enduser
perspective, it is all text, the only difference is a constraint in length,
which does not explain the naming of this fields very well anyways (unless
you are coming from a SQL background where you might be used to that
distinction). As for the UI issues: A TextField with an optional max_length
could use a <input> as default if max_length is specified and <textarea>
otherwise (with a boolean flag to switch that too). Sadly, if we switch
from max_length=None to max_length=xxx, this would mean an alteration of
the database type on most databases, I am not sure if Django can handle
that currently.

I hope this clears it up a little bit,
Florian
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0239cd7a-ca2a-43eb-976c-3d8adf5e456a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Shai Berger
2016-02-27 15:43:46 UTC
Permalink
Hi Florian, Wim and all,
Post by Florian Apolloner
Hi Wim,
Post by Wim Feijen
Can you please elaborate why the current situation is better?
I wouldn't say it is better, just equally good/bad ;)
I disagree.
Post by Florian Apolloner
Post by Wim Feijen
In practice, most people add max_length=255 everytime by hand. I value
your opinion but I fail to understand your reasoning. Could you please
explain further (if you have time)?
In practice, that would mean most people do not care about the length, so
they should be using a TextField, no?
Not really (see below).
Post by Florian Apolloner
(See next paragraph for even another
option) If you want something which works on multiple databases, 255 is a
nonsensical default anyways -- with all the love for emojis out there,
you'd be limiting yourself to 191 (thank MySQL for that 767/4 = 191). What
worries me even more, is that changing this "default" (to accommodate for a
new database for instance) will require database migrations, so all of a
sudden upgrading django will result in migrations for all of your apps.
I think this is why we should be a little more cautious and go for 128 or 100,
rather than "the current maximum all-backend-compatible value".
Post by Florian Apolloner
That all said, if it were for me I'd deprecate CharField and only support
TextField and give it an optional max_length (and a way to switch between
TextArea and LineEdit for UI) -- but that is probably just because on
Postgres there is no fundamental difference between them. From an enduser
perspective, it is all text, the only difference is a constraint in length,
No, that's not the only difference. There's also an understanding that
CharField's are one-liners, that is, there is an implicit limitation (usually
only enforced by UI) that newlines are not allowed in the text.

And that makes it a different kind of beast, in general. For example, I don't
think there are many CharField's whose contents are interpreted as markup. On
the other hand, you don't see many TextField's with Regex validators. In a
user perspective, these are really different data types. I find your suggestion
analogous to a suggestion to remove IntegerField, because "it's just a
FloatField with a constraint on scale".

My 2 cents,
Shai.
Florian Apolloner
2016-02-27 21:57:19 UTC
Permalink
Hi Shai,
Post by Shai Berger
I think this is why we should be a little more cautious and go for 128 or 100,
rather than "the current maximum all-backend-compatible value".
Which somewhat speaks for my point, cause we do not know if the next
backend will need less than 100 chars (But I do agree that this is not
likely). Tough I think that 100 (or 191 for matter) is a ridiculously low
limit. In that sense, allowing max_length=None is probably the safest
solution -- even though we do not know when database backends will crash on
this, but we do not know this for text fields either currently.
Post by Shai Berger
That all said, if it were for me I'd deprecate CharField and only support
Post by Florian Apolloner
TextField and give it an optional max_length (and a way to switch
between
Post by Florian Apolloner
TextArea and LineEdit for UI) -- but that is probably just because on
Postgres there is no fundamental difference between them. From an
enduser
Post by Florian Apolloner
perspective, it is all text, the only difference is a constraint in
length,
No, that's not the only difference. There's also an understanding that
CharField's are one-liners, that is, there is an implicit limitation (usually
only enforced by UI) that newlines are not allowed in the text.
This is a leaky abstraction at best in my opinion. The "understanding" is
mostly coined by the documentation in the sense that it mentions <textarea>
for TextField and <input> for CharField. So it is kind of illogical to not
allow a max_length on TextField. Even with model.full_clean there is no
restriction aside from the length, in that sense a CharField behaves
completely the same as TextField (database issues with index lengths etc
excluded).

For example, I don't
Post by Shai Berger
think there are many CharField's whose contents are interpreted as markup.
Maybe, maybe not, I do not see how this makes a difference here though.
(Same goes for Regex validation).
Post by Shai Berger
In a user perspective, these are really different data types.
Which can be dangerous on it's own, I'd have to check, but I am pretty sure
that nothing prevents the user from submitting newlines in a CharField.
Post by Shai Berger
I find your suggestion
analogous to a suggestion to remove IntegerField, because "it's just a
FloatField with a constraint on scale".
I'd suggest a DecimalField cause I do not like FloatFields either, but yeah
;)

Do not get me wrong, I somewhat agree with everything you say (in the end I
think it is just a matter of taste) -- but I also think that explicit is
better than implicit. From a security point of view, I agree that length
validation should be done whenever it makes sense [which is basically
always, there is no point in allowing to post the maximum length of a
TextField, especially not if it contains markup which you have to render
again ;)], so a CharField with a default max_length seems like a good idea.
On the other hand I think that it will make the distinction for new users
even harder. Assume we change the CharField to have a default max_length,
the first (and probably only thing) users will notice is that CharField is
single line. I am somewhat reading #django 24/7 and can tell you that no
matter how much we write in the docs, users are reading them less and less.
So for all practical purposes a single field allowing for all of that is
imo the best abstraction.

Cheers,
Florian
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/df712817-b19e-411d-b810-2a9bdd42d904%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Shai Berger
2016-02-28 08:32:01 UTC
Permalink
Post by Aymeric Augustin
Hi Shai,
Post by Shai Berger
I think this is why we should be a little more cautious and go for 128 or 100,
rather than "the current maximum all-backend-compatible value".
Which somewhat speaks for my point, cause we do not know if the next
backend will need less than 100 chars (But I do agree that this is not
likely). Tough I think that 100 (or 191 for matter) is a ridiculously low
limit.
No, it isn't, if you think of the default char-field as a one-liner; even in
code, we don't like lines longer than 120 chars... and the default should be
friendly to backends with odd limitations, like Oracle's 8KB limitation on the
sum of inline fields in a row (which means the "max acceptable for oracle
charfield" value, which takes half of that, is a horrible default).
Post by Aymeric Augustin
In that sense, allowing max_length=None is probably the safest
solution -- even though we do not know when database backends will crash on
this, but we do not know this for text fields either currently.
The SQL Standard has many deficiencies, but it's the best resource to work
against in this regard; and the only unlimited-length character field it
supports are LOBs (see [1]). While PG treats LOBs and char-fields the same, it
is the exception rather than the rule, and I suspect it is part of the reason
for why, when push comes to shove, it does not perform as well as the
proprietary databases (don't get me wrong: I prefer it to any of them -- but
it's a bit like comparing Python with C; you pay for expressivity with
performance).
Post by Aymeric Augustin
Post by Shai Berger
Post by Florian Apolloner
That all said, if it were for me I'd deprecate CharField and only
support TextField and give it an optional max_length (and a way to
switch between TextArea and LineEdit for UI) -- but that is probably
just because on Postgres there is no fundamental difference between them.
From an enduser perspective, it is all text, the only difference is a
constraint in length,
No, that's not the only difference. There's also an understanding that
CharField's are one-liners, that is, there is an implicit limitation (usually
only enforced by UI) that newlines are not allowed in the text.
This is a leaky abstraction at best in my opinion. The "understanding" is
mostly coined by the documentation in the sense that it mentions <textarea>
for TextField and <input> for CharField. So it is kind of illogical to not
allow a max_length on TextField.
I am not opposed to a max_length on TextField. I am opposed to setting the
default string field in Django to an unlimited-length one.
Post by Aymeric Augustin
Even with model.full_clean there is no
restriction aside from the length, in that sense a CharField behaves
completely the same as TextField (database issues with index lengths etc
excluded).
Databse issues with LOBs are plenty. PG has spoiled you :) Other databases do
not do functional dependencies properly, and as a result, when you include a
LOB in a grouping or uniqune query you get either database errors (Oracle) or
terrible performance (everyone else except PG, AFAIK).
Post by Aymeric Augustin
For example, I don't
Post by Shai Berger
think there are many CharField's whose contents are interpreted as markup.
Maybe, maybe not, I do not see how this makes a difference here though.
(Same goes for Regex validation).
The point was: They are regarded as different data types, have different
operations etc.
Post by Aymeric Augustin
Post by Shai Berger
In a user perspective, these are really different data types.
Which can be dangerous on it's own, I'd have to check, but I am pretty sure
that nothing prevents the user from submitting newlines in a CharField.
I, too would be surprised if something does. Agreed on "leaky abstraction".
Post by Aymeric Augustin
Post by Shai Berger
I find your suggestion
analogous to a suggestion to remove IntegerField, because "it's just a
FloatField with a constraint on scale".
I'd suggest a DecimalField cause I do not like FloatFields either, but yeah
;)
Do not get me wrong, I somewhat agree with everything you say (in the end I
think it is just a matter of taste) -- but I also think that explicit is
better than implicit.
I tend to agree with original posters in this thread -- explicit here just
means "unnecessary room for error". Django is opinionated about database
design, and I think we stand to gain by expressing on opinion about the
default length of a "regular, one-line charfield".
Post by Aymeric Augustin
From a security point of view, I agree that length
validation should be done whenever it makes sense [which is basically
always, there is no point in allowing to post the maximum length of a
TextField, especially not if it contains markup which you have to render
again ;)], so a CharField with a default max_length seems like a good idea.
On the other hand I think that it will make the distinction for new users
even harder. Assume we change the CharField to have a default max_length,
the first (and probably only thing) users will notice is that CharField is
single line. I am somewhat reading #django 24/7 and can tell you that no
matter how much we write in the docs, users are reading them less and less.
So for all practical purposes a single field allowing for all of that is
imo the best abstraction.
Now, *that* is a leaky abstraction -- good luck hiding all the behavior
differences between limited-length and unlimited-length charfields, on any
backend but PG.

Shai.
Luke Plant
2016-02-28 08:57:17 UTC
Permalink
Replying to this and the other emails in the thread:

Django should not be settling low arbitrary limits for the sake of a database I'm not even using, that's just crazy. Limits as high as 120 are not "big enough for anyone", and will cause problems. (I can give lots of examples). Maximum field length is a critical value that must not be left to some default, because as soon as you come to interoperate with other systems, it *does* matter.


So, this needs to be explicit.

We also need to cover the case of unlimited length Charfield for the databases that support it. This can be covered with a simple sentinel value UNLIMITED_LENGTH or similar. This is a 4th option not listed below.

We could also potentially add another sentinel like DB_MAX_UNICODE_SAFE_LENGTH that works as you expect and is clearly documented, for the sake of 3rd party apps, and comes with the caveat that it produces different behaviour on different databases.

Explicit is better than implicit etc., and in this case there is simply no sensible default behaviour that will cover all our use cases.

Luke
Post by Wim Feijen
Hi guys,
I'd like to reach consensus about
https://code.djangoproject.com/ticket/14094
1. Adding a default max_length = None, which may in practice differ per
database backend.
2. Adding a default max_length of a value which can be used across
databases. Lengths which have been mentioned are: 100, 128, 191 and 255
characters and 4000 bytes.
3. Keep everything as is, being as explicit as possible.
Option 1 does not allow for apps and data to be reused across several apps
and several core developers do not support this idea.
I'm in favour of option 2, because it makes for shorter code and more
importantly, it helps programmers choose a value which actually can be
reused across multiple databases (I didn't know about the possible limit of
191 characters on mysql! Did you?).
I dislike how it is now (option 3), and I don't think it is beneficial here
to force people to be explicit, because people are now using arbitrary
values anyway. 255 is commonly used, the poll app of the Django
tutorial
uses 200. Actually I believe it is a big mess in practice, I see defaults
of 100, 200, 255, 300, 1000 seemingly randomly used. For example, try
searching for CharField in your favourite apps or CMSes.
In addition, adding a default for max_length still allows to be
explicit
when it matters. And it makes life simple when it doesn't matter, and
simple is better than complex.
What would be a good default then? I am in favour of making it as big as
possible while still working on all databases, which would either be 191 of
255 in order to support mysql.
Wim
Post by Tom Christie
I'm with Tom here.
Forcing `max_length` to always be set on CharField feels like the
right
Post by Tom Christie
decision. Having a default there seems unnecessary obscure, and more
likely
Post by Tom Christie
to lead to untested/unnoticed failure cases.
It *could* be that we'd allow `max_length=None` to explicitly turn
off the
Post by Tom Christie
validation, but I'm not sure if that's better than forcing developers
who
Post by Tom Christie
want unvalidated lengths to use TextField.
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
To post to this group, send email to
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/a2cfad14-4384-4118-bc28-d89099433ca5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/EBD26DDE-B91E-4CB6-950F-8E67B98AB97E%40cantab.net.
For more options, visit https://groups.google.com/d/optout.
Florian Apolloner
2016-02-28 10:23:50 UTC
Permalink
Post by Luke Plant
We could also potentially add another sentinel like
DB_MAX_UNICODE_SAFE_LENGTH that works as you expect and is clearly
documented, for the sake of 3rd party apps, and comes with the caveat that
it produces different behaviour on different databases.
This is simply not an option as someone (I think Aymeric) already pointed
out. Technically we do not know DB_MAX_UNICODE_SAFE_LENGTH till the call of
model.save (since you can supply a using argument there), this means forms
will never know the max length and could not perform any validation.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b744bca5-e02e-411e-b9b6-971213aafaa3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loïc Bistuer
2016-02-28 10:32:57 UTC
Permalink
I don't think this is a problem, we could validate that the backend supports it during save then blow up if we detect it doesn't. I think we do that for truncation on MySQL. If the model specifies something that the db doesn't support it's a configuration problem, not a user validation problem.

Sent from my iPhone
Post by Luke Plant
We could also potentially add another sentinel like DB_MAX_UNICODE_SAFE_LENGTH that works as you expect and is clearly documented, for the sake of 3rd party apps, and comes with the caveat that it produces different behaviour on different databases.
This is simply not an option as someone (I think Aymeric) already pointed out. Technically we do not know DB_MAX_UNICODE_SAFE_LENGTH till the call of model.save (since you can supply a using argument there), this means forms will never know the max length and could not perform any validation.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b744bca5-e02e-411e-b9b6-971213aafaa3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5F8A3D88-9CF4-4800-BA65-61D658EB34F2%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Florian Apolloner
2016-02-28 10:50:42 UTC
Permalink
Post by Loïc Bistuer
I don't think this is a problem, we could validate that the backend
supports it during save then blow up if we detect it doesn't. I think we do
that for truncation on MySQL. If the model specifies something that the db
doesn't support it's a configuration problem, not a user validation
problem.
I do not see how blowing up on save is a good idea (especially not if we
ran model validation before). We do not do that for MySQL (at least not if
your project is configured properly, leaving issues with utf8mb4 out for
now) since our model form validation is able to use the supplied max_length
to inform the user that the data is invalid __before__ save.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/23ce2d62-45ed-48f7-9fc1-bc1eb4c47ef8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loïc Bistuer
2016-02-28 14:38:40 UTC
Permalink
There is a precedent to this (although it seems to happen at startup time rather than save time), we loop through all the connections and check that max_length has a value supported by their backend: https://github.com/django/django/blob/master/django/db/backends/mysql/validation.py.

I don’t think we need to create a new sentinel value to mean “unlimited” when `max_length=None` works well semantically, if we want to keep the `max_length` value explicit we can just use `NOT_PROVIDED` as the kwarg default value. It is backwards compatible since `max_length=None` currently issues an error.

Regarding the switch to a default value of 120 or so, I don’t have a strong opinion against it, but I think it’s an orthogonal problem to supporting unlimited varchar on backends that can handle it.
I do not see how blowing up on save is a good idea (especially not if we ran model validation before). We do not do that for MySQL (at least not if your project is configured properly, leaving issues with utf8mb4 out for now) since our model form validation is able to use the supplied max_length to inform the user that the data is invalid __before__ save.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/663F6FA1-1D09-4A21-928A-836157386253%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Luke Plant
2016-02-28 19:06:46 UTC
Permalink
Regarding custom sentinels for values vs custom sentinels for missing values:

The added complication in this case is migrations, and migrations that support multiple Django versions, as is common in 3rd party apps. I haven't thought through the implications, but it is vital to do so, because getting it wrong can seriously hurt the adoption of a feature, and even the adoption of a new version of Django.

It may not be an option to say "for compatibility generate migrations using the lowest supported Django versions", because sometimes old Django versions have bugs that prevent migration generation.

We also need to consider the potential need for several different sentinel values e.g. "unlimited" vs "database defined max length".

Luke
Post by Loïc Bistuer
There is a precedent to this (although it seems to happen at startup
time rather than save time), we loop through all the connections and
https://github.com/django/django/blob/master/django/db/backends/mysql/validation.py.
I don’t think we need to create a new sentinel value to mean
“unlimited” when `max_length=None` works well semantically, if we want
to keep the `max_length` value explicit we can just use `NOT_PROVIDED`
as the kwarg default value. It is backwards compatible since
`max_length=None` currently issues an error.
Regarding the switch to a default value of 120 or so, I don’t have a
strong opinion against it, but I think it’s an orthogonal problem to
supporting unlimited varchar on backends that can handle it.
On Feb 28, 2016, at 6:50 PM, Florian Apolloner
I do not see how blowing up on save is a good idea (especially not if
we ran model validation before). We do not do that for MySQL (at least
not if your project is configured properly, leaving issues with utf8mb4
out for now) since our model form validation is able to use the
supplied max_length to inform the user that the data is invalid
__before__ save.
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
To post to this group, send email to
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/663F6FA1-1D09-4A21-928A-836157386253%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/F5159BD5-23CD-46EF-92A7-EF10BCCE7890%40cantab.net.
For more options, visit https://groups.google.com/d/optout.
Josh Smeaton
2016-02-28 23:59:53 UTC
Permalink
This discussion started because a user wants to leave off max_length for
postgres, which allows the max_length to be omitted right? Is there
anything wrong with `from django.contrib.postgres.fields import CharField`?
The postgres specific CharField will allow max_length to be optional
without interfering with other database backends, confusing the
implementation of the standard CharField, or having to make seemingly
difficult or arbitrary decisions on what a default max_length could be.

FWIW, I'm starting to come around to a default max_length of 128 for the
base CharField, mainly for the reasons Shai has described (mysql max byte
length for unique indexes, 128 characters being enough for the majority of
CharField use-cases). There definitely are some issues with *requiring*
users to arbitrarily define a max_length without having knowledge of all
the ways that 255 may be a problem across backends, or understanding why
255 is even the theoretical max. If we were to consider this, I'd err on
the side of changing CharField to default to max_length=128 rather than
providing a new type.

Cheers
Post by Luke Plant
The added complication in this case is migrations, and migrations that
support multiple Django versions, as is common in 3rd party apps. I haven't
thought through the implications, but it is vital to do so, because getting
it wrong can seriously hurt the adoption of a feature, and even the
adoption of a new version of Django.
It may not be an option to say "for compatibility generate migrations
using the lowest supported Django versions", because sometimes old Django
versions have bugs that prevent migration generation.
We also need to consider the potential need for several different sentinel
values e.g. "unlimited" vs "database defined max length".
Luke
On 28 February 2016 14:38:40 GMT+00:00, "Loïc Bistuer" <
Post by Loïc Bistuer
There is a precedent to this (although it seems to happen at startup time rather than save time), we loop through all the connections and check that max_length has a value supported by their backend: https://github.com/django/django/blob/master/django/db/backends/mysql/validation.py.
I don’t think we need to create a new sentinel value to mean “unlimited” when `max_length=None` works well semantically, if we want to keep the `max_length` value explicit we can just use `NOT_PROVIDED` as the kwarg default value. It is backwards compatible since `max_length=None` currently issues an error.
Regarding the switch to a default value of 120 or so, I don’t have a strong opinion against it, but I think it’s an orthogonal problem to supporting unlimited varchar on backends that can handle it.
I do not see how blowing up on save is a good idea (especially not if we ran model validation before). We do not do that for MySQL (at least not if your project is configured properly, leaving issues with utf8mb4 out for now) since our model form validation is able to use the supplied max_length to inform the user that the data is invalid __before__ save.
--
Sent from my phone
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/02818c88-5b1c-41ee-be86-b48df0580a24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Aymeric Augustin
2016-02-29 09:19:02 UTC
Permalink
Hello,

This thread is getting long. It contains lots of valid arguments. In the interest of seeking consensus, here’s a tentative summary.


Problem

Typing `foo = models.CharField(max_length=100)` gets old quickly. Can we skip the `max_length` argument in many cases?


Constraints

Character limits and their consequences on performance vary widely across databases, which makes it impossible for Django to provide a “large enough and fast enough for everyone” default.

The lowest limit mentioned in the thread is 191 on MySQL. No one talked about third-party backends; we can’t be sure that this value is compatible with unique indexing on all current backends.

Making the default limit backend-dependent sounds like a bad idea, especially for pluggable applications. The implementation also looks complicated compared to the expected benefits.

PostgreSQL is a bit of an outlier. As far as I understand, there’s no performance difference VARCHAR(N) and VARCHAR(). The former just validates the length of the data.

The argument for not providing a default max_length is to have developers think about their data. Unfortunately, in practice, most just write `models.CharField(max_length=255)` and move one.

(FWIW this limit is either a cargo-cult from the 90s <http://stackoverflow.com/a/1217518> or a micro-optimization for MySQL <http://dev.mysql.com/doc/refman/5.7/en/char.html>.)


Consensus (?)

In order to resolve this discussion, I think we must split it in two questions.


1) Should Django provide a reasonable default for CharField.max_length?

It seems to me that the resistance to this idea weakened as the discussion unfolded. Perhaps it’s a losing fight in the long run anyway


A default value could be justified as the “reasonable length for a line of text”. Considering that the optimal line length <https://www.google.com/search?q=optimal+line+length> is around 66 characters, 100 could be a good limit. 120 would work as well.

It seems more future-proof and less debatable not to tie this limit to a technical limitation of the current version of a particular database engine.

On one hand, the default value matters because it will be widely used. On the other hand it doesn’t matter because developers who care about the length of their CharFields should set them explicitly.


2) How can we make it easy for PostgreSQL users to just use VARCHAR()?

Since this is a PostgreSQL-specific feature, having a variant of CharField in django.contrib.postgres that supports and perhaps even defaults to unlimited length shouldn’t be controversial.


I hope this helps!
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/A2F207B6-42EF-4E7E-80B8-6FE4C6FD8014%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Shai Berger
2016-02-29 20:58:05 UTC
Permalink
Hi,

Thank you, Aymeric, for summing up the discussion this way. The division into
two separate problems is indeed required, and I fully support the idea of
setting max_length's default to 100 or 120.
Post by Aymeric Augustin
2) How can we make it easy for PostgreSQL users to just use VARCHAR()?
Since this is a PostgreSQL-specific feature, having a variant of CharField
in django.contrib.postgres that supports and perhaps even defaults to
unlimited length shouldn’t be controversial.
The first -- I believe it was raised very early on by Christophe Pettus -- is
that Django already has a field that manifests on PG as VARCHAR(), and that is
TextField. However, I don't like the idea that PG users should be using
TextField(widget=TextInput) as a replacement for CharField; I find that
counter-intuitive -- even if just because it is a "bad name". Names are
important.

The second -- in response to a comment made by Josh Smeaton -- is that having
django.db.models.CharField with default max_lenth=N (for some finite N) and
django.contrib.postgres.CharField with default max_length=None (meaning
infinity) sounds like a bad idea.
Post by Aymeric Augustin
I hope this helps!
I'm certain it did!

Shai.
Cristiano Coelho
2016-03-01 00:00:29 UTC
Permalink
I find that using TextField rather than CharField just to make postgres use
varchar() is a terrible idea, if you are implementing an reusable app and
it is used on a backend like MySQL where TextFields are created as text
columns which are horribly inneficient and should be avoided at any cost
you will have a really bad time.
I'm not sure about postgres but I want to believe that using varchar
without limits has also some performance considerations that should be
taken care of.
Post by Shai Berger
Hi,
Thank you, Aymeric, for summing up the discussion this way. The division into
two separate problems is indeed required, and I fully support the idea of
setting max_length's default to 100 or 120.
Post by Aymeric Augustin
2) How can we make it easy for PostgreSQL users to just use VARCHAR()?
Since this is a PostgreSQL-specific feature, having a variant of
CharField
Post by Aymeric Augustin
in django.contrib.postgres that supports and perhaps even defaults to
unlimited length shouldn’t be controversial.
The first -- I believe it was raised very early on by Christophe Pettus -- is
that Django already has a field that manifests on PG as VARCHAR(), and that is
TextField. However, I don't like the idea that PG users should be using
TextField(widget=TextInput) as a replacement for CharField; I find that
counter-intuitive -- even if just because it is a "bad name". Names are
important.
The second -- in response to a comment made by Josh Smeaton -- is that having
django.db.models.CharField with default max_lenth=N (for some finite N) and
django.contrib.postgres.CharField with default max_length=None (meaning
infinity) sounds like a bad idea.
Post by Aymeric Augustin
I hope this helps!
I'm certain it did!
Shai.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/37cd6e9c-ec5e-4a69-99a5-84ca11853afe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Markus Holtermann
2016-03-01 01:11:33 UTC
Permalink
From what I understand you will have a hard time doing that at all:

On PG could go with a 'max_length=None' in a 3rd party app. But that
wouldn't be supported on any other database backend. Which means you're
limiting your app to PG. On the other hand you could make your app database
independent by using a TextField which has pretty much the same performance
on PG as a VARCHAR() and is available on other backends. But may perform
bad on other backends. But there's no way to have a text of unlimited
length on MySQL other than a TEXT column.

TL;DR: you can't achieve both ways anyway. Neither now nor if we change the
behavior of Django's max_length attribute on CharFields.

/Markus
Post by Cristiano Coelho
I find that using TextField rather than CharField just to make postgres
use varchar() is a terrible idea, if you are implementing an reusable app
and it is used on a backend like MySQL where TextFields are created as text
columns which are horribly inneficient and should be avoided at any cost
you will have a really bad time.
I'm not sure about postgres but I want to believe that using varchar
without limits has also some performance considerations that should be
taken care of.
Post by Shai Berger
Hi,
Thank you, Aymeric, for summing up the discussion this way. The division into
two separate problems is indeed required, and I fully support the idea of
setting max_length's default to 100 or 120.
Post by Aymeric Augustin
2) How can we make it easy for PostgreSQL users to just use VARCHAR()?
Since this is a PostgreSQL-specific feature, having a variant of
CharField
Post by Aymeric Augustin
in django.contrib.postgres that supports and perhaps even defaults to
unlimited length shouldn’t be controversial.
The first -- I believe it was raised very early on by Christophe Pettus -- is
that Django already has a field that manifests on PG as VARCHAR(), and that is
TextField. However, I don't like the idea that PG users should be using
TextField(widget=TextInput) as a replacement for CharField; I find that
counter-intuitive -- even if just because it is a "bad name". Names are
important.
The second -- in response to a comment made by Josh Smeaton -- is that having
django.db.models.CharField with default max_lenth=N (for some finite N) and
django.contrib.postgres.CharField with default max_length=None (meaning
infinity) sounds like a bad idea.
Post by Aymeric Augustin
I hope this helps!
I'm certain it did!
Shai.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/71e5e743-29bb-4dba-9bf4-6948ab2d1fa9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Marc Tamlyn
2016-03-04 20:05:31 UTC
Permalink
Voting:

1) should there be a default?

I'm lazy, so I would be happy to have one. Where validation matters you can
change it, where it doesn't you don't have to. I'd draw an analogy to
(Positive)(Small)IntegerField - I often use the normal one when I mean some
variant of validation, or some other max/min value.

2) how to make it easy to have no limit on PG?

If it's truly pg specific, then a contrib.pg field seems fine, though
probably call it UnlimitedCharField for lack of ambiguity.

M
Post by Markus Holtermann
On PG could go with a 'max_length=None' in a 3rd party app. But that
wouldn't be supported on any other database backend. Which means you're
limiting your app to PG. On the other hand you could make your app database
independent by using a TextField which has pretty much the same performance
on PG as a VARCHAR() and is available on other backends. But may perform
bad on other backends. But there's no way to have a text of unlimited
length on MySQL other than a TEXT column.
TL;DR: you can't achieve both ways anyway. Neither now nor if we change
the behavior of Django's max_length attribute on CharFields.
/Markus
Post by Cristiano Coelho
I find that using TextField rather than CharField just to make postgres
use varchar() is a terrible idea, if you are implementing an reusable app
and it is used on a backend like MySQL where TextFields are created as text
columns which are horribly inneficient and should be avoided at any cost
you will have a really bad time.
I'm not sure about postgres but I want to believe that using varchar
without limits has also some performance considerations that should be
taken care of.
Post by Shai Berger
Hi,
Thank you, Aymeric, for summing up the discussion this way. The division into
two separate problems is indeed required, and I fully support the idea of
setting max_length's default to 100 or 120.
Post by Aymeric Augustin
2) How can we make it easy for PostgreSQL users to just use VARCHAR()?
Since this is a PostgreSQL-specific feature, having a variant of
CharField
Post by Aymeric Augustin
in django.contrib.postgres that supports and perhaps even defaults to
unlimited length shouldn’t be controversial.
The first -- I believe it was raised very early on by Christophe Pettus -- is
that Django already has a field that manifests on PG as VARCHAR(), and that is
TextField. However, I don't like the idea that PG users should be using
TextField(widget=TextInput) as a replacement for CharField; I find that
counter-intuitive -- even if just because it is a "bad name". Names are
important.
The second -- in response to a comment made by Josh Smeaton -- is that having
django.db.models.CharField with default max_lenth=N (for some finite N) and
django.contrib.postgres.CharField with default max_length=None (meaning
infinity) sounds like a bad idea.
Post by Aymeric Augustin
I hope this helps!
I'm certain it did!
Shai.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/71e5e743-29bb-4dba-9bf4-6948ab2d1fa9%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/71e5e743-29bb-4dba-9bf4-6948ab2d1fa9%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMwjO1Gj3E4_83qdbQmQwXe_WdTXUf7yuZ44-aQ9jgvJN9-5sg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loïc Bistuer
2016-03-05 00:24:17 UTC
Permalink
I’m not too keen on a contrib.pg field. CharField is the base class of many fields, a __init__ kwarg approach such as max_length=None allows us to reach those as well.
Post by Marc Tamlyn
1) should there be a default?
I'm lazy, so I would be happy to have one. Where validation matters you can change it, where it doesn't you don't have to. I'd draw an analogy to (Positive)(Small)IntegerField - I often use the normal one when I mean some variant of validation, or some other max/min value.
2) how to make it easy to have no limit on PG?
If it's truly pg specific, then a contrib.pg field seems fine, though probably call it UnlimitedCharField for lack of ambiguity.
M
On PG could go with a 'max_length=None' in a 3rd party app. But that wouldn't be supported on any other database backend. Which means you're limiting your app to PG. On the other hand you could make your app database independent by using a TextField which has pretty much the same performance on PG as a VARCHAR() and is available on other backends. But may perform bad on other backends. But there's no way to have a text of unlimited length on MySQL other than a TEXT column.
TL;DR: you can't achieve both ways anyway. Neither now nor if we change the behavior of Django's max_length attribute on CharFields.
/Markus
I find that using TextField rather than CharField just to make postgres use varchar() is a terrible idea, if you are implementing an reusable app and it is used on a backend like MySQL where TextFields are created as text columns which are horribly inneficient and should be avoided at any cost you will have a really bad time.
I'm not sure about postgres but I want to believe that using varchar without limits has also some performance considerations that should be taken care of.
Hi,
Thank you, Aymeric, for summing up the discussion this way. The division into
two separate problems is indeed required, and I fully support the idea of
setting max_length's default to 100 or 120.
Post by Aymeric Augustin
2) How can we make it easy for PostgreSQL users to just use VARCHAR()?
Since this is a PostgreSQL-specific feature, having a variant of CharField
in django.contrib.postgres that supports and perhaps even defaults to
unlimited length shouldn’t be controversial.
The first -- I believe it was raised very early on by Christophe Pettus -- is
that Django already has a field that manifests on PG as VARCHAR(), and that is
TextField. However, I don't like the idea that PG users should be using
TextField(widget=TextInput) as a replacement for CharField; I find that
counter-intuitive -- even if just because it is a "bad name". Names are
important.
The second -- in response to a comment made by Josh Smeaton -- is that having
django.db.models.CharField with default max_lenth=N (for some finite N) and
django.contrib.postgres.CharField with default max_length=None (meaning
infinity) sounds like a bad idea.
Post by Aymeric Augustin
I hope this helps!
I'm certain it did!
Shai.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/71e5e743-29bb-4dba-9bf4-6948ab2d1fa9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMwjO1Gj3E4_83qdbQmQwXe_WdTXUf7yuZ44-aQ9jgvJN9-5sg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/95635BD4-26EE-4948-A45A-CECFE83A501C%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Shai Berger
2016-03-05 13:12:43 UTC
Permalink
Post by Loïc Bistuer
I’m not too keen on a contrib.pg field. CharField is the base class of many
fields, a __init__ kwarg approach such as max_length=None allows us to
reach those as well.
That's a good point; Can we enable max_length=None in a mixin?

Shai Berger
2016-02-28 11:35:46 UTC
Permalink
Post by Florian Apolloner
Post by Luke Plant
We could also potentially add another sentinel like
DB_MAX_UNICODE_SAFE_LENGTH that works as you expect and is clearly
documented, for the sake of 3rd party apps, and comes with the caveat
that it produces different behaviour on different databases.
This is simply not an option as someone (I think Aymeric) already pointed
out. Technically we do not know DB_MAX_UNICODE_SAFE_LENGTH till the call of
model.save (since you can supply a using argument there), this means forms
will never know the max length and could not perform any validation.
...or it could just mean we take the minimum of all defined databases (we would
probably like to ask the routers for the db_for_write, but this requires the
model to be defined already, so we can't do that).

No, this is still a problem: During the development, you decide to add a
second database, and all of the sudden get migrations for the whole project,
because the minimum over all defined databases changed. No good. So, I'd modify
it to something like

django.db.max_unicode_safe_length(database=DEFAULT_DB_ALIAS)

However, as I explained earlier, DB_MAX_UNICODE_SAFE_LENGTH is not a suitable
default for max_length, so this is a little besides the point here.
Post by Florian Apolloner
Django should not be settling low arbitrary limits for the sake of a
database I'm not even using, that's just crazy. Limits as high as 120 are
not "big enough for anyone"
Defaults are not limits, and I don't think anybody setting a default expects
that it would fit everyone. If we did, we'd consider removing the argument
completely.
Post by Florian Apolloner
Maximum field length is a critical value that must not be left
to some default, because as soon as you come to interoperate with other
systems, it *does* matter.
So, this needs to be explicit.
Erg, again -- nobody is suggesting to make users unable to set their own
max_length; but many Django projects actually do not need interaction with
other systems. These are probably the majority of projects, and even a bigger
part of the projects done by beginners. We can remove an obstacle by saying,
"unless you have reason to think otherwise, we suggest that 120 is a good
length for a field intended to hold one line of text". It is an obstacle both
because beginners these days are not used to think of length limits (since the
programming language has a string type), and because they are quite likely to
get it wrong (either make it too short, or go over MySql's limit of 191).

So, to summarise: I think the database-relative constants you proposed are a
good idea; but I don't think any of them fits the bill of a default, and I do
think a default will do more good than harm.

Shai.
Continue reading on narkive:
Loading...