Moving database backends out of the core

[Michael Manfre]

Full disclosure, I maintain django-mssql and am biased toward having all
database backends treated as if they were 3rd party backends.

The current discussion about "Switch to database-level autocommit" (
http://bit.ly/ZlVERI) reminded me of a past discussion regarding moving the
database backends out of the core. I don't remember exactly where I heard
the discussion, but I'd like to revive it. My proposal is to move all
database backends out of Django, except for an sqlite reference
implementation.

Django has already done a similar transition with
django.contrib.localflavor, which provides us with some of the basic
answers about whether this sort of thing is possible and a rough guide
about how to make it happen.


Benefits
======
- One of the biggest benefits of this change would be that database
backends will no longer be bound to Django's release cycle and Django would
not need to make a dot release for a database specific issue.

- This will result in Django having a more flexible/robust database API.
Any of the existing complexity imposed on the core by a specific database
backend could be abstracted out to the API.

- In some instances this will simplify the core's code by pushing the
complexity in to the specific database backend.

Concerns
=======

These are the current issues/concerns faced by 3rd party database backends.

- All backends would need to support multiple versions of Django, or at
least properly document which versions of Django the backend works with

- People will need to (pip) install the database backend that they want
for each project

- Less popular database backends may get left behind

- Each project will have its own tests and will also need to run against
the entire Django test suite

Migration Path:
===========

I believe this can be accomplished without needing a multi-version
deprecation cycle because unlike localflavors, there should be very little,
if any, user code that imports a specific database backend directly. The
only instances I can think of are the DATABSE_ENGINE path and non standard
features that I've added to django-mssql; custom field types, non-standard
aggregates, etc. If a multi-version deprecation cycle is needed, it will be
possible to fix the import paths as localflavors.

Django 1.6 or 1.7
- Create separate github project for each database backend and move code
- Abstract out any backend specific code in the ORM by extending the
database API
- Move backend specific tests to appropriate project

Regards,
Michael Manfre

[Florian Apolloner]

Hi,

Post by Michael Manfre
The current discussion about "Switch to database-level autocommit" (
http://bit.ly/ZlVERI) reminded me of a past discussion regarding moving
the database backends out of the core. I don't remember exactly where I
heard the discussion, but I'd like to revive it. My proposal is to move all
database backends out of Django, except for an sqlite reference
implementation.

While I agree that moving database adapters out of core has some merit, I
don't think that having sqlite as a reference implementation is a good
idea: For one some features are somewhat hacky in sqlite (and people tend
to copy from reference implementations, so it should be as clean as
possible) and it's lack of data validation makes it imo a nogo. The
reference implementation should imo also have strong support for GIS, which
is somewhat okay on sqlite but quite hard to install. So if we were to do
that I'd either vote for postgres or supporting postgres and sqlite inside
of core (the later solely for fast tests).

Cheers,
Florian

[Michael Manfre]

Post by Florian Apolloner
Hi,
While I agree that moving database adapters out of core has some merit, I
don't think that having sqlite as a reference implementation is a good
idea: For one some features are somewhat hacky in sqlite (and people tend
to copy from reference implementations, so it should be as clean as
possible)

I think I made a mistake in using the word "reference". "Included" would be
a more accurate definition of its intended status. Ideally, no database
backends would be in the core, but I realize that is not entirely practical
because of the need to run tests. Sqlite also provides the lowest barrier
for new users to get through the tutorial.

Post by Florian Apolloner
From my experience of maintaining django-mssql, I usually have to reference

all of the backends depending on the issue I'm working on. I start by
looking at whichever database provides the closest behavioral match and
then compare it to the others. Every database is different, so it's not
like copy & paste is going to happen. Database backends are also not
something a large number of people will run out and create. I imagine that
even with Postgres, mysql, and oracle in separate projects they will still
be maintained by a collection of developers that will most likely include
core devs.

Post by Florian Apolloner
and it's lack of data validation makes it imo a nogo.

The lack of data validation is definitely a nogo for production sites, but
imo sqlite in production is also a nogo.

The reference implementation should imo also have strong support for GIS,

Post by Florian Apolloner
which is somewhat okay on sqlite but quite hard to install. So if we were
to do that I'd either vote for postgres or supporting postgres and sqlite
inside of core (the later solely for fast tests).

. I've never tried to install GIS for sqlite, but is the difficulty due to
lack of documentation or just sheer number of steps?

Regards,
Michael

[Florian Apolloner]

Hi,

Post by Michael Manfre
The lack of data validation is definitely a nogo for production sites, but
imo sqlite in production is also a nogo.

Right, but shipping Django with a non production db might send interesting
signals to endusers ;)

The reference implementation should imo also have strong support for GIS,

Post by Michael Manfre

Post by Florian Apolloner
which is somewhat okay on sqlite but quite hard to install. So if we were
to do that I'd either vote for postgres or supporting postgres and sqlite
inside of core (the later solely for fast tests).

. I've never tried to install GIS for sqlite, but is the difficulty due to
lack of documentation or just sheer number of steps?

Well it's not to bad, we did document it after all, but it usually requires
recompiling sqlite and pysqlite (most importantly you can't pull it from
your distros repos).

Regards,
Florian

[Michael Manfre]

Post by Florian Apolloner

Post by Michael Manfre
The lack of data validation is definitely a nogo for production sites,
but imo sqlite in production is also a nogo.

Right, but shipping Django with a non production db might send interesting
signals to endusers ;)

It might, but it's all about marketing. "Django, now 10% less monolithic!"

Regards,
Michael

[VernonCole]

Huh?? I am running sqlite with GIS on my laptop right now. It wasn't THAT
hard to install. Yes, the documentation could use some cleanup, but it got
me through the tutorial okay and gives me a platform to learn GIS on.
I really support the idea of sqlite and postgres in the core and moving
everything else outside.
(Not that my support counts for much.)

Post by Florian Apolloner
Hi,

Post by Michael Manfre
The lack of data validation is definitely a nogo for production sites,
but imo sqlite in production is also a nogo.

Right, but shipping Django with a non production db might send interesting
signals to endusers ;)
The reference implementation should imo also have strong support for GIS,

Post by Michael Manfre

Post by Florian Apolloner
which is somewhat okay on sqlite but quite hard to install. So if we were
to do that I'd either vote for postgres or supporting postgres and sqlite
inside of core (the later solely for fast tests).

. I've never tried to install GIS for sqlite, but is the difficulty due
to lack of documentation or just sheer number of steps?

Well it's not to bad, we did document it after all, but it usually
requires recompiling sqlite and pysqlite (most importantly you can't pull
it from your distros repos).
Regards,
Florian

[Shai Berger]

Hi,

Post by Michael Manfre
Full disclosure, I maintain django-mssql and am biased toward having all
database backends treated as if they were 3rd party backends.

In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

Post by Michael Manfre
Benefits
======
- One of the biggest benefits of this change would be that database
backends will no longer be bound to Django's release cycle and Django would
not need to make a dot release for a database specific issue.

I think you are missing something crucial: Django depends on its commonly-used
backends. A critical problem with the Postgresql or MySql backend is a
critical problem for the Django project, regardless of separation to
repositories. And where the dependency is weaker, being in core was not a
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

So, IMO, you can't escape the need to have a set of "blessed" backends, which
core is committed to their correct functioning, and thus need to be, at least
to some degree, in sync with the Django main project.

Post by Michael Manfre
- This will result in Django having a more flexible/robust database API.
Any of the existing complexity imposed on the core by a specific database
backend could be abstracted out to the API.
- In some instances this will simplify the core's code by pushing the
complexity in to the specific database backend.

These benefits are, IMO, orthogonal to the separation you suggest. I can't
speak for core, but I doubt that a proposed change to make the database API
more robust and flexible will be rejected just because it is originally
motivated by a 3rd-party, rather than core, backend.

Post by Michael Manfre
Concerns
=======
- Less popular database backends may get left behind

I am worried that the result of such change will not be that MSSQL is treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

Post by Michael Manfre
===========
I believe this can be accomplished without needing a multi-version
deprecation cycle because unlike localflavors, there should be very little,
if any, user code that imports a specific database backend directly.

South certainly does, I'm sure it's not the only one.

Shai.

[Florian Apolloner]

Hi Shai,

Post by Shai Berger
In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

I am obviously biased against postgres as my previous post indicated, but
regardless of that I think that MSSQL should stay outside of core. No
core-developer I know actually uses Windows as base OS and to my knowledge
no one uses MSSQL. This would put the expected support for South below what
we have for Oracle currently.

And where the dependency is weaker, being in core was not a

Post by Shai Berger
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

Now that we test on Oracle once again I am committed to have a full passing
testsuite with 1.6 (whatever that means, since I don't use Oracle on a
daily base I can't say what a running testsuite actually means for a
project).

I am worried that the result of such change will not be that MSSQL is

Post by Shai Berger
treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

That might as well be true, if Oracle is outside of core I personally
wouldn't put much effort in fixing stuff there (this obviously might change
depending on whether I use Oracle at work or not).

Cheers,
Florian

[Florian Apolloner]

Post by Florian Apolloner
I am obviously biased against postgres as my previous post indicated, but
regardless of that I think that MSSQL should stay outside of core. No
core-developer I know actually uses Windows as base OS and to my knowledge
no one uses MSSQL. This would put the expected support for South below what
we have for Oracle currently.

s/South/MSSQL sry

[Andre Terra]

Hi Florian,

Post by Florian Apolloner
I am obviously biased against postgres as my previous post indicated, but
regardless of that I think that MSSQL should stay outside of core. No
core-developer I know actually uses Windows as base OS and to my knowledge
no one uses MSSQL. This would put the expected support for South below what
we have for Oracle currently.

If I may weigh in on the matter as an outsider, if we consider "The Django
Project" as a "business", insofar as it aims to have as many users and be
as ubiquitous as possible, there's considerable value in having MSSQL
included in core. If it were up to me, I would develop on Linux using
postgres and host everything on Heroku, but at work I'm restricted to
corporate rules, MS SQL or Oracle, and Windows.

I have a strong feeling that by embracing Windows as a first-class platform
for developing and running Django, we're enabling current and future users
to easily integrate the framework in existent business infrastructure,
which can only be seen as a good thing for the project.


Cheers,
AT

[Florian Apolloner]

Hi Andre,

but at work I'm restricted to corporate rules, MS SQL or Oracle, and
Windows.

Right, that's probably one of the reasons why Oracle is in core (aside from
the fact that we were completely monolithic at that time).

I have a strong feeling that by embracing Windows as a first-class platform

for developing and running Django, we're enabling current and future users
to easily integrate the framework in existent business infrastructure,
which can only be seen as a good thing for the project.

You are obviously right that it would help, but my point about not having
any core developers in that area still stands and makes it hard to support
something like that. FWIW, I think that supporting Windows would be far
easier than supporting MSSQL. Looking at the bug tracker we have a few bugs
for windows (some are false positives):
https://code.djangoproject.com/query?status=assigned&status=new&description=~windows&col=id&col=summary&col=status&col=owner&col=type&col=component&order=priority

Cheers,
Florian

[Marc Tamlyn]

Seems to me that if database backends were separate from Django then the
postgres backend would get a long way ahead of the others as much as the
other backends would get behind - it's bound to attract the most work and
be adding custom fields like hstore, arrays, json... I think this would be
great for most of my use cases, but then again I also have to deploy a
small proportion of sites on MySQL, so it's bound to bite me.

That said, I don't know why Oracle is included and MSSQL is not - pretty
much none of the core devs currently use either (or would recommend it
given the choice). Cross compatibility of Django core is really important,
and should ideally be as far reaching as we can achieve. I'd rather see
MSSQL pulled in, and hopefully testing of it via Travis at some point than
things split out, as I think it's better for Django.

[Jacob Kaplan-Moss]

I don't know why Oracle is included and MSSQL is not [...]

It's essentially because a group of people made a concerted effort to
get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
and then committed to maintaining it. Lately the original people who'd
made that commitment have dropped off a bit, but there's always been
enough attention to keep it up to date.

If someone -- preferably a group -- stepped up and committed to
keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
and giving those people commit access to keep it up to date.

[This is me speaking personally, not as a BDFL. It'd have to be a
discussion, not just my fiat.]

Jacob

[Alex Ogier]

It's worth mentioning that Django appears in the Python Tools for Visual
Studio (it's one of the default project templates)[1]. There's recent
introductory material on deploying Django applications to Windows Azure[2]
and Windows Server 2008[3], though it seems the older material at least
explicitly recommends MySQL and not SQL Server. I don't know if this is the
kind of thing that leads to demand among Windows folks for Windows support
for Django, and if any significant fraction of them want to use MSSQL with
Django, but Django is certainly becoming mainstream for them.

[1]: http://pytools.codeplex.com/
[2]:
http://www.windowsazure.com/en-us/develop/python/tutorials/django-with-visual-studio/
[3]:
http://www.windowsazure.com/en-us/develop/python/tutorials/web-app-with-django-and-mysql/

Best,
Alex Ogier

Post by Jacob Kaplan-Moss

I don't know why Oracle is included and MSSQL is not [...]

It's essentially because a group of people made a concerted effort to
get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
and then committed to maintaining it. Lately the original people who'd
made that commitment have dropped off a bit, but there's always been
enough attention to keep it up to date.
If someone -- preferably a group -- stepped up and committed to
keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
and giving those people commit access to keep it up to date.
[This is me speaking personally, not as a BDFL. It'd have to be a
discussion, not just my fiat.]
Jacob
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers?hl=en
.
For more options, visit https://groups.google.com/groups/opt_out.

[Erik Romijn]

Hi all,

It seems to me we are mixing a few different discussions here:
1. Should Django core have as few database backends as possible?
1a. If yes, which ones should stay in Django core?
2. What should we do, if anything, with the current situation where
it seems difficult to guarantee the quality of the Oracle backend?
3. Should a MSSQL Django backend be included in Django core?

Regarding question 1, I feel we should keep a fair set of backends,
like the current one, included in Django core.

Yes, we could move them to separate projects. All the same arguments
apply here as why it's not necessarily good for a project to end up
in contrib. However, in theory any component of Django could be moved
out of core and be independently maintained. The whole ORM, the
templating system, the CSRF protection, and so on. I haven't seen any
reason why we should move the MySQL backend out, but not the templating
system.

Besides that, the close integration and development of all these parts
is exactly the reason I like Django. When I download a version of Django,
or upgrade, I know that all components in there will work well together.
I can run my site on any of the supported databases, and they will all
work together with the provided admin. I can build forms on the models
I build with the ORM. The forms will nicely fit in with the templates.

When I started using Django, I had looked at several alternatives which
were more modular. However, they required me to make tons of choices,
each involving numerous consequences. If I picked ORM A, I could have
databases X, Y and Z, but no admin. For the admin I had to use ORM B,
but that did not support database Z or many to many fields. And so on.
When I tried Django, it was a relief to see I could download a single
package, and everything would just work together.

If we move database backends out of core, my big concern is that
the same will happen there. This is fine for many components we use
in many Django projects - but not for something as fundamental as the
database backend. When the admin doesn't work on any of the officially
supported backends, that should be a release blocker for Django itself,
because I think it will be a blocker for the users.

Regarding question 2, Oracle support, I think a great step forward has
been made with the addition of Oracle to the Django continuous
integration setup. That alone should help us improve the consistent
quality of Oracle support.

cheers,
Erik

[Alex Ogier]

Here's something I've been thinking about:

As a rule, assuming that backends are not bug-riddled and do not have
needlessly shoddy performance, nearly all of the value of an ORM is in the
set of frontend features it exposes to application developers. Given a
certain frontend API, then the only value in iterating the backend
independently from the frontend is to fix uncaught bugs, or to improve
performance, there's no value in adding features.

Also as a rule, adding features to a frontend independently from the
backend has little value. Without an implementation in whatever backend you
are on, it only serves as a guide for future backend development, so you
might as well push for backends to catch up in the same release cycle. (Or
the feature doesn't require backend support, but then it doesn't hurt to
reversion the backend anyways).

So basically, there's not much value in independently versioning and
maintaining frontends and backends to the ORM. This is in contrast to, say,
localflavor, where an improvement in Mongolian localization can immediately
help every Mongolian website in every Django version. So this primary
motivation for independent maintenance is not a factor in the ORM. Hence I
think the core team should include as many backends as it can handle in
core (where "handle" means test that they function properly for each
release). Oracle had been slipping, but from what I understand it's now in
the CI server and back to passing most tests.

So I see no reason to split off backends unless the maintenance burden is
such that they are chronically broken in every Django release. And every
reason to add MSSQL unless the maintenance burden is too high.

Best,
Alex Ogier

Post by Erik Romijn
Hi all,
1. Should Django core have as few database backends as possible?
1a. If yes, which ones should stay in Django core?
2. What should we do, if anything, with the current situation where
it seems difficult to guarantee the quality of the Oracle backend?
3. Should a MSSQL Django backend be included in Django core?
Regarding question 1, I feel we should keep a fair set of backends,
like the current one, included in Django core.
Yes, we could move them to separate projects. All the same arguments
apply here as why it's not necessarily good for a project to end up
in contrib. However, in theory any component of Django could be moved
out of core and be independently maintained. The whole ORM, the
templating system, the CSRF protection, and so on. I haven't seen any
reason why we should move the MySQL backend out, but not the templating
system.
Besides that, the close integration and development of all these parts
is exactly the reason I like Django. When I download a version of Django,
or upgrade, I know that all components in there will work well together.
I can run my site on any of the supported databases, and they will all
work together with the provided admin. I can build forms on the models
I build with the ORM. The forms will nicely fit in with the templates.
When I started using Django, I had looked at several alternatives which
were more modular. However, they required me to make tons of choices,
each involving numerous consequences. If I picked ORM A, I could have
databases X, Y and Z, but no admin. For the admin I had to use ORM B,
but that did not support database Z or many to many fields. And so on.
When I tried Django, it was a relief to see I could download a single
package, and everything would just work together.
If we move database backends out of core, my big concern is that
the same will happen there. This is fine for many components we use
in many Django projects - but not for something as fundamental as the
database backend. When the admin doesn't work on any of the officially
supported backends, that should be a release blocker for Django itself,
because I think it will be a blocker for the users.
Regarding question 2, Oracle support, I think a great step forward has
been made with the addition of Oracle to the Django continuous
integration setup. That alone should help us improve the consistent
quality of Oracle support.
cheers,
Erik
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers?hl=en
.
For more options, visit https://groups.google.com/groups/opt_out.

[Russell Keith-Magee]

Post by Alex Ogier
As a rule, assuming that backends are not bug-riddled and do not have
needlessly shoddy performance, nearly all of the value of an ORM is in the
set of frontend features it exposes to application developers. Given a
certain frontend API, then the only value in iterating the backend
independently from the frontend is to fix uncaught bugs, or to improve
performance, there's no value in adding features.
Also as a rule, adding features to a frontend independently from the
backend has little value. Without an implementation in whatever backend you
are on, it only serves as a guide for future backend development, so you
might as well push for backends to catch up in the same release cycle. (Or
the feature doesn't require backend support, but then it doesn't hurt to
reversion the backend anyways).
So basically, there's not much value in independently versioning and
maintaining frontends and backends to the ORM. This is in contrast to, say,
localflavor, where an improvement in Mongolian localization can immediately
help every Mongolian website in every Django version. So this primary
motivation for independent maintenance is not a factor in the ORM. Hence I
think the core team should include as many backends as it can handle in
core (where "handle" means test that they function properly for each
release). Oracle had been slipping, but from what I understand it's now in
the CI server and back to passing most tests.
So I see no reason to split off backends unless the maintenance burden is
such that they are chronically broken in every Django release. And every
reason to add MSSQL unless the maintenance burden is too high.

And that last clause is the catch.

I completely agree with Jacob's analysis of the status quo, and I agree
largely with his position on having MSSQL in the core.

I'd have no problem seeing MSSQL in the core - it's at least as high
profile as Oracle, and although the demand for a MSSQL backend is
restricted to a particular subset of the community, there is at least
*some* demand for it to exist. Having MSSQL in core would allow us to hold
our head high and say we support Microsoft platforms. Microsoft is even a
member of the DSF at this point, so they're at least notionally interested
in Django as a platform.

The maintenance burden is the problem. Historically, we've already had to
*remove* a MSSQL backend because of bit rot (see [1]) - this isn't a
pattern I want to repeat in the future. I don't want to add a backend to
core unless I'm *certain* that it will see long term maintenance.

[1] https://github.com/django/django/commit/c30a050e41

There is, however, a possible middle ground, following the example set by
Flask: we introduce to Django a list of "officially recognised" extensions.
These extensions are still maintained as external projects, but the core
team promise to include these projects as part of the testing regimen for a
release. If the MSSQL backend were recognised like this, we would run the
full Django test suite against the MSSQL backend, and if that testing
revealed a bug in the test suite which was identified as being due to a
change in Django's codebase, that bug would be a release blocker (Bugs in
the backend itself would still be the backend's responsibility, and not
release blocking on Django)

Looking outside database backends, this could also apply to high-profile
projects like haystack, django-registration, and so on. This would also
serve to highlight projects in our community that are 'defacto standards',
or good examples of reusability, without needing to expand the core team or
the size of contrib, and show that the core project is explicitly
interested in the broader ecosystem.

Yours,
Russ Magee %-)

[Michael Manfre]

Post by Russell Keith-Magee
I completely agree with Jacob's analysis of the status quo, and I agree
largely with his position on having MSSQL in the core.
I'd have no problem seeing MSSQL in the core - it's at least as high
profile as Oracle, and although the demand for a MSSQL backend is
restricted to a particular subset of the community, there is at least
*some* demand for it to exist. Having MSSQL in core would allow us to hold
our head high and say we support Microsoft platforms. Microsoft is even a
member of the DSF at this point, so they're at least notionally interested
in Django as a platform.

/slightly off topic
I didn't realize they were a DSF member. I have been wanting to set up
Jenkins to run the test suite on Windows with the ultimate goal of getting
its results added to ci.djangoproject.com. Knowing that Microsoft is a DSF
member will probably help gain access to a VM on Azure to make that a
reality. I've so far managed to get an introduction to a PM on one of the
Azure teams to help get a more direct answer to some Django, Windows, and
SQL problems that have been raised to my attention.

/back on topic

The maintenance burden is the problem. Historically, we've already had to

Post by Russell Keith-Magee
*remove* a MSSQL backend because of bit rot (see [1]) - this isn't a
pattern I want to repeat in the future. I don't want to add a backend to
core unless I'm *certain* that it will see long term maintenance.
[1] https://github.com/django/django/commit/c30a050e41

The demand for MSSQL would probably increase if it were included in core
because companies could be more confident about its ongoing support.

There are three options for using MSSQL with Django; django-pyodbc,
django-mssql, or write your own. The third option is not really a viable
option for most users.

django-pyodbc is cross platform, but not updated very often and has limited
stored procedure support. The official repo on Google code was last updated
in 2011, but there is a github fork with its last commit 8 months ago.

django-mssql is actively maintained and will be for at least the next few
years because it's used for my employer's production site that is critical
to business operations. The backend also supports stored procedures almost
to the same extent as a Windows business application. The only real
downside to the backend is that it only works on Windows, reducing its
usefulness to a much smaller, but growing, minority of Django users.

If Django wants to "hold [its] head high and support Microsoft platforms",
then I'm willing and capable of helping out with MSSQL and Windows support.
The only real caveat is that my employer needs me to maintain a certain
level of stored procedure functionality, so if MSSQL were included in the
core by using django-pyodbc instead of django-mssql, I would only be able
to commit personal time toward maintaining it, instead of also being able
to set aside business hours.

There is, however, a possible middle ground, following the example set by

Post by Russell Keith-Magee
Flask: we introduce to Django a list of "officially recognised" extensions.
These extensions are still maintained as external projects, but the core
team promise to include these projects as part of the testing regimen for a
release. If the MSSQL backend were recognised like this, we would run the
full Django test suite against the MSSQL backend, and if that testing
revealed a bug in the test suite which was identified as being due to a
change in Django's codebase, that bug would be a release blocker (Bugs in
the backend itself would still be the backend's responsibility, and not
release blocking on Django)

This is sort of the idea I had in mind for backends while I was typing up
this proposal. You articulated it better than I did. A main problem I now
see with it is if the ownership of a bug cannot be mutually agreed upon
between Django and the extension, then this could cause doubt about what it
really means to be "officially recognized".

Regards,
Michael Manfre

Since DjangoCon, I now hear the email author's voice when reading some of
these emails.

[VernonCole]

I would love to volunteer for the project of MSSQL support. It happens
that django-mssql uses a fork of my adodbapi v2.1 code. I pulled those
changes back into the main branch of adodbapi (and did a lot of useful
refactoring thanks to the good work done on the fork) at version 2.3.
Apparently the announcement of that did not get to the right people, since
the peek I just took shows the forked code still in use.

The two major obstacles I can see to MSSQL support (either as part of the
core or as a supported add-on) are:
1) lack of an MSSQL server and test bed which the maintainers can use on a
regular basis.
2) django-mssql is not Python 3 ready.

The first of those obstacles has already been addressed.

The second is fairly easy. I suspect (not having examined the code) that
most of the Python3 problems will be found in the old fork of adodbapi. If
we simply change django-mssql to call the current version of adodbapi all
of that goes away. The current version is already present on their Python
library, I am sure, because it ships as part of pywin32, which is also a
prerequisite for the V2.1 code. Yes, all of pywin32, including adodbapi,
runs on Python 3 -- has done for years.

Would there be a "market" for a supported version of django which runs
MSSQL? Does the bear live in the woods??? Take, for example, my present
employer. (I escaped from the "we are a strictly Microsoft shop" employer
where I used to work.) We here at eHealth Africa use Ubuntu and postgres as
much as possible. But it's not always possible. For example, part of the
project I am working on now will use ArcGIS -- which only runs on Windows.
Then one of our funding organizations (ever heard of the Gates Foundation?)
required us to host a database on an MSSQL server. We bit the bullet,
bought the licenses (it's their money, after all) and set up the SQL
server. Do you think we want to use django on it? Huh?

If it will help kill polio, we will use anybody's software.

As to the question of moving MSSQL into the core, or moving the two Oracle
dbs out of it (you do remember who owns MySQL?) -- I would suggest pulling
all non-FOSS out. Would you accept my patches to the core? Of course not.
and you should not: I don't know django well enough. Would you accept my
patches to an MSSQL back end? I should hope so.

There has been talk. from time to time, about extending django to have more
multi-database capability. The changes that would be needed to run three
different db engines from outside the core would easily extend to other
engines. People tend to think of ADO as only talking to Microsoft
databases. Nothing could be farther from the truth. When maintaining
adodbapi, I normally test against MS-SQL Server, Microsoft "Jet" (a.k.a.
ACCESS), MySQL, and postgres. I have also personally used it to get data
from IBM DB2, an Active Directory server, and a .csv file. If some idiot
has written an odbc driver for it, I will let you read it from Python.

Build the hooks.
--
Vernon Cole

On Thu, Mar 7, 2013 at 8:13 PM, Russell Keith-Magee <

Post by Russell Keith-Magee
I completely agree with Jacob's analysis of the status quo, and I agree
largely with his position on having MSSQL in the core.
I'd have no problem seeing MSSQL in the core - it's at least as high
profile as Oracle, and although the demand for a MSSQL backend is
restricted to a particular subset of the community, there is at least
*some* demand for it to exist. Having MSSQL in core would allow us to hold
our head high and say we support Microsoft platforms. Microsoft is even a
member of the DSF at this point, so they're at least notionally interested
in Django as a platform.

/slightly off topic
I didn't realize they were a DSF member. I have been wanting to set up
Jenkins to run the test suite on Windows with the ultimate goal of getting
its results added to ci.djangoproject.com. Knowing that Microsoft is a
DSF member will probably help gain access to a VM on Azure to make that a
reality. I've so far managed to get an introduction to a PM on one of the
Azure teams to help get a more direct answer to some Django, Windows, and
SQL problems that have been raised to my attention.
/back on topic
The maintenance burden is the problem. Historically, we've already had to

Post by Russell Keith-Magee
*remove* a MSSQL backend because of bit rot (see [1]) - this isn't a
pattern I want to repeat in the future. I don't want to add a backend to
core unless I'm *certain* that it will see long term maintenance.
[1] https://github.com/django/django/commit/c30a050e41

The demand for MSSQL would probably increase if it were included in core
because companies could be more confident about its ongoing support.
There are three options for using MSSQL with Django; django-pyodbc,
django-mssql, or write your own. The third option is not really a viable
option for most users.
django-pyodbc is cross platform, but not updated very often and has
limited stored procedure support. The official repo on Google code was last
updated in 2011, but there is a github fork with its last commit 8 months
ago.
django-mssql is actively maintained and will be for at least the next few
years because it's used for my employer's production site that is critical
to business operations. The backend also supports stored procedures almost
to the same extent as a Windows business application. The only real
downside to the backend is that it only works on Windows, reducing its
usefulness to a much smaller, but growing, minority of Django users.
If Django wants to "hold [its] head high and support Microsoft platforms",
then I'm willing and capable of helping out with MSSQL and Windows support.
The only real caveat is that my employer needs me to maintain a certain
level of stored procedure functionality, so if MSSQL were included in the
core by using django-pyodbc instead of django-mssql, I would only be able
to commit personal time toward maintaining it, instead of also being able
to set aside business hours.
There is, however, a possible middle ground, following the example set by

Post by Russell Keith-Magee
Flask: we introduce to Django a list of "officially recognised" extensions.
These extensions are still maintained as external projects, but the core
team promise to include these projects as part of the testing regimen for a
release. If the MSSQL backend were recognised like this, we would run the
full Django test suite against the MSSQL backend, and if that testing
revealed a bug in the test suite which was identified as being due to a
change in Django's codebase, that bug would be a release blocker (Bugs in
the backend itself would still be the backend's responsibility, and not
release blocking on Django)

This is sort of the idea I had in mind for backends while I was typing up
this proposal. You articulated it better than I did. A main problem I now
see with it is if the ownership of a bug cannot be mutually agreed upon
between Django and the extension, then this could cause doubt about what it
really means to be "officially recognized".
Regards,
Michael Manfre
Since DjangoCon, I now hear the email author's voice when reading some of
these emails.

[Javier Guerra Giraldez]

People tend to think of ADO as only talking to Microsoft databases. Nothing
could be farther from the truth. When maintaining adodbapi, I normally test
against MS-SQL Server, Microsoft "Jet" (a.k.a. ACCESS), MySQL, and postgres.
I have also personally used it to get data from IBM DB2, an Active Directory
server, and a .csv file. If some idiot has written an odbc driver for it, I
will let you read it from Python.

nice.

does the client run on non-win32 plaforms? from a cursory read of the
website, adodbapi seems to require pywin32.

[Michael Manfre]

On Fri, Mar 8, 2013 at 9:33 AM, Javier Guerra Giraldez

Post by VernonCole

People tend to think of ADO as only talking to Microsoft databases.

Nothing

could be farther from the truth. When maintaining adodbapi, I normally

test

against MS-SQL Server, Microsoft "Jet" (a.k.a. ACCESS), MySQL, and

postgres.

I have also personally used it to get data from IBM DB2, an Active

Directory

server, and a .csv file. If some idiot has written an odbc driver for

it, I

will let you read it from Python.

nice.
does the client run on non-win32 plaforms? from a cursory read of the
website, adodbapi seems to require pywin32.

adodbapi is included with pywin32 and requires it.

Regards,
Michael Manfre

[VernonCole]

[My reply is below, after the question... scan down. VDC]

Post by VernonCole

People tend to think of ADO as only talking to Microsoft databases.

Nothing

could be farther from the truth. When maintaining adodbapi, I normally

test

against MS-SQL Server, Microsoft "Jet" (a.k.a. ACCESS), MySQL, and

postgres.

I have also personally used it to get data from IBM DB2, an Active

Directory

server, and a .csv file. If some idiot has written an odbc driver for

it, I

will let you read it from Python.

nice.
does the client run on non-win32 plaforms? from a cursory read of the
website, adodbapi seems to require pywin32.
--
Javier

Sorry about terse nature of the website information. I need to repent and
write better documentation.

ADO is a strictly Microsoft animal. It seems that Microsoft invented ODBC,
which was quickly recognized as such a great idea that everyone adopted
it. So they had to invent something "better", and that was supposed to be
ADO. As it happens, the most redeeming feature of ADO is that in runs in
ODBC mode by default. That having been said, it is very powerful, and does
have the ability to reach out to almost any form of tabular data. But it
does so using COM calls, which are very proprietary.

I have been worrying over the problem of how to read an ADO dataset from
Linux for years. I was hoping that IronPython and Mono would provide the
answer, but it turns out that ADO.NET also uses COM, and looses some of the
power (like timely update of .rowcount) in the process. It does not even
work well in a pure Windows environment. I have abandoned that code fork.
The open source odbc products leave things to be desired, too, the largest
being that they only work in ODBC mode and cannot switch to ADO mode when
the connection string requires it. I have begun to think that eGenix is on
to something with their "mxODBC connect" product: they use a connection
server.

Here's how it might be done: Somewhere, there would have to be a Windows
box running an ADO connection server program. Perhaps it could be on the
actual database server, perhaps not. A copy of adodbapi which notices that
it is running where os.name != 'nt' would bring up a link to the connection
server (using RPC?) which would do the actual data access and pass the
results back. A simple server would be trivial to write. One which worked
well under a heavy thread load might not be so easy, but my thinking is
that a heavily used django server which made frequent use of an MSSQL
database would already be running on a Windows box itself and would not
need any connection server.

If there is interest, I will start coding a prototype.

[Florian Apolloner]

Hi,

Post by Michael Manfre
django-mssql is actively maintained and will be for at least the next few
years because it's used for my employer's production site that is critical
to business operations. The backend also supports stored procedures almost
to the same extent as a Windows business application. The only real
downside to the backend is that it only works on Windows, reducing its
usefulness to a much smaller, but growing, minority of Django users.

What's the state of http://code.google.com/p/pymssql/ ? If we have MSSQL in
core I'd really like to be able to talk with it from a Linux machine too,
it would also make testing easier since we'd just need a VBox with MSSQL ;)
Supporting a commercially available product but requiring to run even more
commercial software seems like counter intuitive to me.

Regards,
Florian

[Michael Manfre]

Post by Florian Apolloner
What's the state of http://code.google.com/p/pymssql/ ?

Their stable release relies upon a no longer supported Microsoft driver and
has some known issues that make it unusable (lack of MARS support being the
biggest issue encountered in production). They've been working toward
switching to FreeTDS, but I have not actively followed their progress since
last summer and don't know how close they are to the switch. I might be
remembering incorrectly, but some of the issues with django-pyodbc's
support of stored procedures might have been rooted in FreeTDS. I haven't
reevaluated django-pyodbc for a few years because of its lack of support
and the native client drivers available to django-mssql are faster.

Post by Florian Apolloner
If we have MSSQL in core I'd really like to be able to talk with it from a
Linux machine too, it would also make testing easier since we'd just need a
VBox with MSSQL ;) Supporting a commercially available product but
requiring to run even more commercial software seems like counter intuitive
to me.

Django needs a VBox running Windows for testing, regardless if MSSQL is in
the core.

Regards,
Michael Manfre

[Florian Apolloner]

Post by Florian Apolloner
If we have MSSQL in core I'd really like to be able to talk with it from a
Linux machine too, it would also make testing easier since we'd just need a
VBox with MSSQL ;) Supporting a commercially available product but
requiring to run even more commercial software seems like counter intuitive
to me.
Django needs a VBox running Windows for testing, regardless if MSSQL is in
the core.

That doesn't mean we actually have to support Microsoft in making MSSQL
only accessible from Windows… Also there is a large difference between
running a vm with a database server and running a vm (read: jenkins slaves)
to execute tests.

If Microsoft wants to use/support Django I guess they could easily publish
a Python db-api 2.x driver. I'll happily code the Django part if the
prerequisites would exist (assuming MSSQL actually does work ;))…

Regards,
Florian

[VernonCole]

Post by Florian Apolloner

Post by Florian Apolloner
If we have MSSQL in core I'd really like to be able to talk with it from
a Linux machine too, it would also make testing easier since we'd just need
a VBox with MSSQL ;) Supporting a commercially available product but
requiring to run even more commercial software seems like counter intuitive
to me.
Django needs a VBox running Windows for testing, regardless if MSSQL is
in the core.

That doesn't mean we actually have to support Microsoft in making MSSQL
only accessible from Windows… Also there is a large difference between
running a vm with a database server and running a vm (read: jenkins slaves)
to execute tests.
If Microsoft wants to use/support Django I guess they could easily publish
a Python db-api 2.x driver. I'll happily code the Django part if the
prerequisites would exist (assuming MSSQL actually does work ;))…
Regards,
Florian

Oh, MSSQL actually works (and works, and works, and works...) and finally
comes up with your data. It may perform rather slowly, but it does
perform. And many of us are forced to use it by events beyond our
control. Most of us would probably prefer a Posix box for our django
server, too, but we don't always have that choice, either. Django servers
are expected to run on Windows, and it would be a Really Good Idea (tm) if
it were tested there during development, no matter which database it is
using.

As for django servers reading MS SQL databases, I can see three use cases:
1) django server running Windows in heavy production. This works fine,
Michael has a pile of them.
2) django server running Posix in a test environment. I think I can make
this work using a connection server -- well enough to handle runsever at
least.
3) django server running Posix in heavy production. There we run into
problems with lightweight support for database drivers. But will this
situation ever exist in real life? I don't think so. I suspect that any
heavy-use shops will also be using Windows to run their django servers.

As for Microsoft coding a db api 2.x driver -- why should they, when a
perfectly good FOSS one already exists (If I humbly say so myself)?
--
Vernon Cole

[Javier Guerra Giraldez]

But will this situation ever exist in real life? I don't think so. I
suspect that any heavy-use shops will also be using Windows to run their
django servers.

to me, that reads "heavy clients of MSSQL have to run windows".

what a pity.

[r***@public.gmane.org]

from thread "Moving database backends out of the core"

There is, however, a possible middle ground, following the example set by Flask: we introduce to Django a list of "officially recognised" extensions. These extensions are still maintained as external projects, but the core team promise to include these projects as part of the testing regimen for a release. If the MSSQL backend were recognised like this, we would run the full Django test suite against the MSSQL backend, and if that testing revealed a bug in the test suite which was identified as being due to a change in Django's codebase, that bug would be a release blocker (Bugs in the backend itself would still be the backend's responsibility, and not release blocking on Django)
Looking outside database backends, this could also apply to high-profile projects like haystack, django-registration, and so on. This would also serve to highlight projects in our community that are 'defacto standards', or good examples of reusability, without needing to expand the core team or the size of contrib, and show that the core project is explicitly interested in the broader ecosystem.

I think this is a great idea. The backend testing makes sense to me (run Django's tests). How does testing sanctioned apps work, though? You run _their_ tests, and if they are caused by bugs in Django then they are release blockers?

I really like the idea, and would like to help if I can. I imagine the leg work will be mostly in setting up the testing infrastructure, although the difficult part will be setting guidelines for and choosing apps that should be in this group.

Ryan

[Russell Keith-Magee]

Post by r***@public.gmane.org
from thread "Moving database backends out of the core"

Post by Russell Keith-Magee
There is, however, a possible middle ground, following the example set

by Flask: we introduce to Django a list of "officially recognised"
extensions. These extensions are still maintained as external projects, but
the core team promise to include these projects as part of the testing
regimen for a release. If the MSSQL backend were recognised like this, we
would run the full Django test suite against the MSSQL backend, and if that
testing revealed a bug in the test suite which was identified as being due
to a change in Django's codebase, that bug would be a release blocker (Bugs
in the backend itself would still be the backend's responsibility, and not
release blocking on Django)

Post by Russell Keith-Magee
Looking outside database backends, this could also apply to high-profile

projects like haystack, django-registration, and so on. This would also
serve to highlight projects in our community that are 'defacto standards',
or good examples of reusability, without needing to expand the core team or
the size of contrib, and show that the core project is explicitly
interested in the broader ecosystem.
I think this is a great idea. The backend testing makes sense to me (run
Django's tests). How does testing sanctioned apps work, though? You run
_their_ tests, and if they are caused by bugs in Django then they are
release blockers?

Caveat: I haven't given a huge amount of thought to this beyond the basic
idea, so this is all my initial thoughts.

I can see three options:

1) We run the tests. This requires that we come up with a standard for
describing how test environments work. This could be as simple as requiring
that python setup.py test works, and produces output in a known format.
Requiring the use of tox, or Jenkins/Travis configs would be another option.

2) We leave running the tests up to individual projects, but require them
to expose test results in a standard way. No idea how this would be
specified, other than possibly a HTTP API endpoint.

3) We make it completely manual. This is something that only needs to
happen around release time, so as long as it's relatively easy to run the
full suite of tests manually, it may be possible to manually maintain a
database of "working/not working".

Option 3 might be the easiest way to get going - after all, you can always
add automation infrastructure later.

I really like the idea, and would like to help if I can. I imagine the leg

Post by r***@public.gmane.org
work will be mostly in setting up the testing infrastructure, although the
difficult part will be setting guidelines for and choosing apps that should
be in this group.

Agreed that this is more a policy/procedures issue than a technical issue.
I'm not sure what the best approach is for this - I'm open to suggestions.

As for how to get this ball rolling -- well, I suppose there's two tasks
that could be addressed immediately:

* A list of candidate packages -- even if it's not the final approved
list, we need somewhere to start

* A first cut at a site to host this information.

My local Django Users Group (in Perth, Western Australia) had some
interesting discussions about this sort of packaging project at our last
meeting; I'm hoping to start knocking together some code at our next
meeting. As soon as we've got something to show off, I'll be sure to do so.

Yours,
Russ Magee %-)

[Rahul]

Hi,
I maintain the IBM DB2 Django backend (ibm_db_django)
project(http://code.google.com/p/ibm-db/). We have been maintaining this
extension since Django-1.0 and have always kept it up-to-date.
The http://code.google.com/p/ibm-db/ project also provides a Python DBI
compliant driver, based on which the Django back-end is built. (For the
record, in the past some of our customers have requested for Django to
include the DB2 backend in core, which didnt happen). I really like
Russell's idea to introduce officially recognized extensions. If that is
the outcome of this thread, I would like to add the DB2 extension as
officially recognized and will help in setting infrastructure for this if I
can. I would prefer the automated test environment for recognized
extensions so that any core Django commit will get expose if it cause any
problem in some specific backend, In this way committer can raise a ticket
against specific backend if any changes required in that.

Thanks,
Rahul Priyadarshi

Post by Russell Keith-Magee

Post by r***@public.gmane.org
from thread "Moving database backends out of the core"

Post by Russell Keith-Magee
There is, however, a possible middle ground, following the example set

by Flask: we introduce to Django a list of "officially recognised"
extensions. These extensions are still maintained as external projects, but
the core team promise to include these projects as part of the testing
regimen for a release. If the MSSQL backend were recognised like this, we
would run the full Django test suite against the MSSQL backend, and if that
testing revealed a bug in the test suite which was identified as being due
to a change in Django's codebase, that bug would be a release blocker (Bugs
in the backend itself would still be the backend's responsibility, and not
release blocking on Django)

Post by Russell Keith-Magee
Looking outside database backends, this could also apply to

high-profile projects like haystack, django-registration, and so on. This
would also serve to highlight projects in our community that are 'defacto
standards', or good examples of reusability, without needing to expand the
core team or the size of contrib, and show that the core project is
explicitly interested in the broader ecosystem.
I think this is a great idea. The backend testing makes sense to me (run
Django's tests). How does testing sanctioned apps work, though? You run
_their_ tests, and if they are caused by bugs in Django then they are
release blockers?

Caveat: I haven't given a huge amount of thought to this beyond the basic
idea, so this is all my initial thoughts.
1) We run the tests. This requires that we come up with a standard for
describing how test environments work. This could be as simple as requiring
that python setup.py test works, and produces output in a known format.
Requiring the use of tox, or Jenkins/Travis configs would be another option.
2) We leave running the tests up to individual projects, but require them
to expose test results in a standard way. No idea how this would be
specified, other than possibly a HTTP API endpoint.
3) We make it completely manual. This is something that only needs to
happen around release time, so as long as it's relatively easy to run the
full suite of tests manually, it may be possible to manually maintain a
database of "working/not working".
Option 3 might be the easiest way to get going - after all, you can always
add automation infrastructure later.
I really like the idea, and would like to help if I can. I imagine the leg

Post by r***@public.gmane.org
work will be mostly in setting up the testing infrastructure, although the
difficult part will be setting guidelines for and choosing apps that should
be in this group.

Agreed that this is more a policy/procedures issue than a technical issue.
I'm not sure what the best approach is for this - I'm open to suggestions.
As for how to get this ball rolling -- well, I suppose there's two tasks
* A list of candidate packages -- even if it's not the final approved
list, we need somewhere to start
* A first cut at a site to host this information.
My local Django Users Group (in Perth, Western Australia) had some
interesting discussions about this sort of packaging project at our last
meeting; I'm hoping to start knocking together some code at our next
meeting. As soon as we've got something to show off, I'll be sure to do so.
Yours,
Russ Magee %-)

[VernonCole]

My organization just hit a use case where we need MS-SQL support.

I am jumping on board, so there are at least two of us who can do
maintenance.

I must say that I would prefer quasi-supported status (akin to admin
and geodjango) rather than actually being in the core. I think it would
be a better fit for most situations. We will always be a small minority of
django users. I would just like some assurance that pull requests needed
to provide good hook support for external database backends got prompt
attention from the core developers.
--
Vernon Cole

Post by Jacob Kaplan-Moss

I don't know why Oracle is included and MSSQL is not [...]

It's essentially because a group of people made a concerted effort to
get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
and then committed to maintaining it. Lately the original people who'd
made that commitment have dropped off a bit, but there's always been
enough attention to keep it up to date.
If someone -- preferably a group -- stepped up and committed to
keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
and giving those people commit access to keep it up to date.
[This is me speaking personally, not as a BDFL. It'd have to be a
discussion, not just my fiat.]
Jacob

[Johannes Hoppe]

I want to address a completely different point, and that *innovation*. I
believe that 3rd party backends could lead to more innovation in the Django
ORM.
Currently if you want to introduce a new feature for your database, you are
faced with a lot of complications added by databases you might not be
familiar with. Furthermore you might be requested to makes those features
available for databases you haven't used before. This drastically increases
the bar for contributing innovative new features. As an example, I wanted
to add database defaults for Postgres and multiple insert return values. I
finished the postgres feature in 2 sprints, but it took me another half
year to implement the same for Oracle. Mainly because I never used Oracle
before.

Beyond that I see that the sheer effort to remove the backends from core
could lead to better design. We currently assume that all databases speak
some flavor of Std SQL, which isn't even true for the currently supported
databases and certainly not for the verify famous mongo-db backend.

In conclusion, more separation will lead to more diversity. But that is a
good thing, something to embrace and can lead to great results. I would
even go as far as to kick Postgres out too, contrary to what Florian
suggested. I believe Postgres could benefit from a separate package.

I think Django has the best ORMs and us being able to make changes and
innovate, can ensure that this is still true a decade from now.

Post by VernonCole
My organization just hit a use case where we need MS-SQL support.
I am jumping on board, so there are at least two of us who can do
maintenance.
I must say that I would prefer quasi-supported status (akin to admin
and geodjango) rather than actually being in the core. I think it would
be a better fit for most situations. We will always be a small minority of
django users. I would just like some assurance that pull requests needed
to provide good hook support for external database backends got prompt
attention from the core developers.
--
Vernon Cole

Post by Jacob Kaplan-Moss

I don't know why Oracle is included and MSSQL is not [...]

It's essentially because a group of people made a concerted effort to
get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
and then committed to maintaining it. Lately the original people who'd
made that commitment have dropped off a bit, but there's always been
enough attention to keep it up to date.
If someone -- preferably a group -- stepped up and committed to
keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
and giving those people commit access to keep it up to date.
[This is me speaking personally, not as a BDFL. It'd have to be a
discussion, not just my fiat.]
Jacob

[Tom Forbes]

I see where you are coming from, but I'm not sure it will have the intended
effect. One of the great things about Django is that for the most part
database features work everywhere. If we split them out there will be more
innovation, sure, but we may loose the 'database transparency' we currently
have leading to inconsistencies.

It may also be harder to coordinate changes across databases that require
changes to core.

On your point about non-standard backends, maybe we should focus on making
it easier to add third party backends and standardize some of the
internals? We could treat the backend as external (i.e no special hacks for
them) but keep them in the Django package. If we make the interfaces a bit
more abstract it would be a lot easier for these backends to exist without
hacks perhaps.

As with everything, it's a trade off. I'm not sure there is a clear answer
or better solution for all cases than we currently have.

Post by Johannes Hoppe
I want to address a completely different point, and that *innovation*. I
believe that 3rd party backends could lead to more innovation in the Django
ORM.
Currently if you want to introduce a new feature for your database, you
are faced with a lot of complications added by databases you might not be
familiar with. Furthermore you might be requested to makes those features
available for databases you haven't used before. This drastically increases
the bar for contributing innovative new features. As an example, I wanted
to add database defaults for Postgres and multiple insert return values. I
finished the postgres feature in 2 sprints, but it took me another half
year to implement the same for Oracle. Mainly because I never used Oracle
before.
Beyond that I see that the sheer effort to remove the backends from core
could lead to better design. We currently assume that all databases speak
some flavor of Std SQL, which isn't even true for the currently supported
databases and certainly not for the verify famous mongo-db backend.
In conclusion, more separation will lead to more diversity. But that is a
good thing, something to embrace and can lead to great results. I would
even go as far as to kick Postgres out too, contrary to what Florian
suggested. I believe Postgres could benefit from a separate package.
I think Django has the best ORMs and us being able to make changes and
innovate, can ensure that this is still true a decade from now.

Post by VernonCole
My organization just hit a use case where we need MS-SQL support.
I am jumping on board, so there are at least two of us who can do
maintenance.
I must say that I would prefer quasi-supported status (akin to admin
and geodjango) rather than actually being in the core. I think it would
be a better fit for most situations. We will always be a small minority of
django users. I would just like some assurance that pull requests needed
to provide good hook support for external database backends got prompt
attention from the core developers.
--
Vernon Cole

Post by Jacob Kaplan-Moss

I don't know why Oracle is included and MSSQL is not [...]

It's essentially because a group of people made a concerted effort to
get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
and then committed to maintaining it. Lately the original people who'd
made that commitment have dropped off a bit, but there's always been
enough attention to keep it up to date.
If someone -- preferably a group -- stepped up and committed to
keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
and giving those people commit access to keep it up to date.
[This is me speaking personally, not as a BDFL. It'd have to be a
discussion, not just my fiat.]
Jacob

--

You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/978f91f8-709a-48dc-b2e2-b940e19a9f7f%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/978f91f8-709a-48dc-b2e2-b940e19a9f7f%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Michael Manfre]

Post by Tom Forbes
On your point about non-standard backends, maybe we should focus on making
it easier to add third party backends and standardize some of the
internals? We could treat the backend as external (i.e no special hacks for
them) but keep them in the Django package. If we make the interfaces a bit
more abstract it would be a lot easier for these backends to exist without
hacks perhaps.

Over the past several years when I first started this thread, the API has
improved and these types of improvements reduced the need to monkey patch
or completely rewrite bits of the ORM in django-mssql. Continuing to
move/keep any DB specific hacks out of the ORM and in to their respective
backend makes Django friendlier for 3rd party database backends. The API is
still deemed internal and if you push for a more official database backend
API, I suggest clearly stating that it will still be exempt from the
standard deprecation policy. There was a lot of strong opposition to
changing that policy in the past.

Regards,
Michael Manfre
--
GPG Fingerprint: 74DE D158 BAD0 EDF8
keybase.io/manfre

[Dan Davis]

I think this would only work if most database specific backends were
maintained by the djangoproject itself, allowing for integration tests that
test compatibility.
To my mind, a strong ORM to backend API is a great thing, but we also need
stronger backend integration tests.

[Johannes Hoppe]

How about we do pretend like we want to remove the backends from core and do the homework and refactor the backends to be less intertwined but keep them in the same repo/package. Maybe that’s how we can have the best of both worlds. It is going to be harder to motivate people to put in the work though :/

I think this would only work if most database specific backends were maintained by the djangoproject itself, allowing for integration tests that test compatibility.
To my mind, a strong ORM to backend API is a great thing, but we also need stronger backend integration tests.
--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/O-g06EM6XMM/unsubscribe.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/23759c70-4148-4ac7-8dfa-ca97f1dae97d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shai Berger]

Hi,

On Mon, 26 Nov 2018 00:57:04 -0800 (PST)

Post by Johannes Hoppe
I want to address a completely different point, and that
*innovation*. I believe that 3rd party backends could lead to more
innovation in the Django ORM.
Currently if you want to introduce a new feature for your database,
you are faced with a lot of complications added by databases you
might not be familiar with. Furthermore you might be requested to
makes those features available for databases you haven't used before.
This drastically increases the bar for contributing innovative new
features. As an example, I wanted to add database defaults for
Postgres and multiple insert return values. I finished the postgres
feature in 2 sprints, but it took me another half year to implement
the same for Oracle. Mainly because I never used Oracle before.

I just wanted to point out that this road to innovation is already
open. There's nothing stopping you from writing a custom database
backend and releasing it to PyPI. Granted, then your new feature does
not become part of Django, but that would also be the case if the
standard backends were not part of Django.

I should point out that this point is not theoretical -- e.g. an
extended Oracle backend existed to support connection pooling[1], and
when I wanted to add database instrumentation[2], I first implemented
it (though I hadn't published that) in a custom backend, inheriting the
existing PG backend.

Shai

[1] https://github.com/JohnPapps/django-oracle-drcp
[2] https://docs.djangoproject.com/en/2.1/topics/db/instrumentation/

[Johannes Hoppe]

Hi Shai,

Thank you. You make a very good point. That is exactly what I meant. I have written small backends myself to add new features. The problem is that those features rarely make it into the core backend, because those are too static inside Django.

Best
-Joe

Post by Shai Berger
Hi,
On Mon, 26 Nov 2018 00:57:04 -0800 (PST)

Post by Johannes Hoppe
I want to address a completely different point, and that
*innovation*. I believe that 3rd party backends could lead to more
innovation in the Django ORM.
Currently if you want to introduce a new feature for your database,
you are faced with a lot of complications added by databases you
might not be familiar with. Furthermore you might be requested to
makes those features available for databases you haven't used before.
This drastically increases the bar for contributing innovative new
features. As an example, I wanted to add database defaults for
Postgres and multiple insert return values. I finished the postgres
feature in 2 sprints, but it took me another half year to implement
the same for Oracle. Mainly because I never used Oracle before.

I just wanted to point out that this road to innovation is already
open. There's nothing stopping you from writing a custom database
backend and releasing it to PyPI. Granted, then your new feature does
not become part of Django, but that would also be the case if the
standard backends were not part of Django.
I should point out that this point is not theoretical -- e.g. an
extended Oracle backend existed to support connection pooling[1], and
when I wanted to add database instrumentation[2], I first implemented
it (though I hadn't published that) in a custom backend, inheriting the
existing PG backend.
Shai
[1] https://github.com/JohnPapps/django-oracle-drcp
[2] https://docs.djangoproject.com/en/2.1/topics/db/instrumentation/
--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/O-g06EM6XMM/unsubscribe.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20181128013339.2901e7a7.shai%40platonix.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Manfre]

Hi Andre,

Post by Andre Terra
If I may weigh in on the matter as an outsider, if we consider "The Django
Project" as a "business", insofar as it aims to have as many users and be
as ubiquitous as possible, there's considerable value in having MSSQL
included in core. If it were up to me, I would develop on Linux using
postgres and host everything on Heroku, but at work I'm restricted to
corporate rules, MS SQL or Oracle, and Windows.

Business rules dictate Windows and MSSQL for me as well. Despite the fact
that including MSSQL in the core would mean it has a better bare minimum
amount of support, I'm strongly opposed to it being included. The core
already has too many databases included.

Post by Andre Terra
I have a strong feeling that by embracing Windows as a first-class
platform for developing and running Django, we're enabling current and
future users to easily integrate the framework in existent business
infrastructure, which can only be seen as a good thing for the project.

Agreed. Windows should be treated as a first-class OS, but I don't think
any of the core devs use Windows as their primary OS. I only know of one
core dev that even runs the Django testsuite on Windows from time to time.

Regards,
Michael

[Michael Manfre]

Post by Florian Apolloner
Hi Shai,

Post by Shai Berger
In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

I am obviously biased against postgres as my previous post indicated, but
regardless of that I think that MSSQL should stay outside of core. No
core-developer I know actually uses Windows as base OS and to my knowledge
no one uses MSSQL. This would put the expected support for South below what
we have for Oracle currently.

I agree that MSSQL should stay out of the core. I believe that adding any
more database in to the core would be a large step in the wrong direction.

Post by Florian Apolloner
And where the dependency is weaker, being in core was not a

Post by Shai Berger
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

Now that we test on Oracle once again I am committed to have a full
passing testsuite with 1.6 (whatever that means, since I don't use Oracle
on a daily base I can't say what a running testsuite actually means for a
project).

Is it not possible to maintain the same standard with it as a separate
project? Automated tests could still easily be accomplished and the only
real difference about the backend working with Django is that the releases
of the projects would not be a direct blocker for each other. 3rd party
backends are only able to provide eventual support. Sometimes the new
behaviors are in place ahead of the Django release, sometimes it lands
shortly after. A huge benefit is that if there is a bug/change in the
postgres backend, it doesn't require another Django release.

Post by Florian Apolloner
I am worried that the result of such change will not be that MSSQL is

Post by Shai Berger
treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

That might as well be true, if Oracle is outside of core I personally
wouldn't put much effort in fixing stuff there (this obviously might change
depending on whether I use Oracle at work or not).

Having a backend outside of core might encourage others to step up and
maintain it, especially if they depend on it. I'm sure many currently have
the mentality of "the cores will take care of it, so I don't have to".

It sounds as if Oracle is already not really a supported database and will
only have a passing test suite. As you stated, "whatever that means, since
I don't use Oracle on a daily base I can't say what a running testsuite
actually means for a project".

Regards,
Michael

[Jonathan Slenders]

Post by Michael Manfre
Full disclosure, I maintain django-mssql and am biased toward having all
database backends treated as if they were 3rd party backends.

The question is, how monolithic do we want to have Django?

If we would move database backends out of core. Why not move everything in
django.db away from core? There are other ORMs for Django. And the same is
true for the templating system.

For me, personally, it's really a big +1 for splitting Django up in several
logical parts, and for removing the global state. (So that we could create
a Django *instance*, and add a postgres *instance* to django ourself.)

I want to to see something like this:

d = Django(
database_backend=Postgres(postgres_settings),
urls=our_url_patters,
templates=TemplateSystem(template_settings),
midldeware=...,
)

No disadvantages I think. But it's not an easy migration path.

Cheers,
Jonathan

[Aymeric Augustin]

Post by Michael Manfre
- Less popular database backends may get left behind

To be honest, I wouldn't make any effort to keep the MySQL and Oracle backends at feature-parity with PostgreSQL and SQLite if they weren't in core.
--
Aymeric.

[Claude Paroz]

As for me, wearing my FOSS-only developper hat, I'd suggest we only include
backends in core for products having an OSI-approved license. And yes, the
Oracle backend is already problematic at this regard. That would mean I'm
-0 to -1 for including any other db backend (or any other component) based
on a proprietary product.

Cheers,
Claude