Discussion:
What The Enterprise wants from Django
(too old to reply)
Jacob Kaplan-Moss
2010-01-19 21:26:17 UTC
Permalink
Hi folks --

I had a conversation this morning with one of my clients. In the
interests of being a good corporate citizen I'll refrain from
mentioning who other than (a) they're big (Fortune 1000), (b) you've
heard of them, and (c) they're using Django.

Before our chat, they'd invited any engineers internally to bitch
about Django, and they shared some of the pain points they'd come
across. I took some notes; the really good news is that it's a short
list, and we already know about most of it.

Before I share the list, I should mention that I'm not arguing we have
to immediately devote resources to these problems, nor even that we
have to solve them. I'm sharing this just as food for thought, and as
a perspective from a class of developers who mostly don't participate
in our process at all.

So:

The main issue they have is multiple database support. Music to my
ears (and especially Alex's :) since it looks like 1.2 will solve most
(all?) of their issues. My main takeaway from this point was that the
best way to please companies like this one will be to ship 1.2 on
time. Which will happen. So we rock.

The next big one was the "startup signal" issue -- they've got lots of
code that needs to run at startup time, and there's no great mechanism
to do that currently. The core devs have talked about this one a *lot*
over the years, and it's obviously a hard one -- for one, there's no
clear definition of what "startup" means -- but a common theme seems
to be that bigger, more complex applications need some way to cleanly
run one-time, expensive startup tasks.

Next, we discussed the difficulty of maintaining large installations
with multiple sites, a plethora of apps, and unspeakable possible
combinations thereof. Django's settings mechanism is simple and easy
to use, but once you have hundreds of sites it starts getting really
tricky to keep things in sync across hundreds of settings files. We
talked at some length about different possibilities for dynamic
settings infrastructure; this would especially come in handy for folks
wanting to build application servers/services like App Engine or the
nascent Toppcloud.

<digress>If you've not yet checked out Toppcloud, do so now. I'll
wait.</digress>

Finally, we ruminated over the difficulties in building rich internet
applications. Sure, writing HTML/CSS/JS/Python/SQL by hand works fine,
but we doesn't really have a good answer for the people who want
something IDE or GUI-ish. Meanwhile, Adobe and Microsoft are putting
all sorts of marketing dollars into Flex/Silverlight, and although
HTML5 can do some amazing things, the lack of tooling is a big danger.
(I've written at more length about this in the past:
http://jacobian.org/writing/snakes-on-the-web/#s-rich-web-applications).

Of course, there may not be much us backend folks can do about this
problem -- I'm certainly not enough of a GUI application guy to be
able to even start to think about this problem -- but the lack of an
end-to-end solution will nonetheless put some off from writing web
applications with open source tools.

So there we are. This is very much a brain dump, and I don't really
expect any concrete action to result from it. However, I found some
really interesting stuff there, and I thought I'd share.

Jacob
David Cramer
2010-01-20 03:59:06 UTC
Permalink
The first three have been huges ones with us. We're just now running
into the settings issue, but would love to see what people can come up
with for solutions (we dont have a good one). Glad to see multi db is
finally shipping, and excited to see what can be done for startup
procs.
Post by Jacob Kaplan-Moss
Hi folks --
I had a conversation this morning with one of my clients. In the
interests of being a good corporate citizen I'll refrain from
mentioning who other than (a) they're big (Fortune 1000), (b) you've
heard of them, and (c) they're using Django.
Before our chat, they'd invited any engineers internally to bitch
about Django, and they shared some of the pain points they'd come
across. I took some notes; the really good news is that it's a short
list, and we already know about most of it.
Before I share the list, I should mention that I'm not arguing we have
to immediately devote resources to these problems, nor even that we
have to solve them. I'm sharing this just as food for thought, and as
a perspective from a class of developers who mostly don't participate
in our process at all.
The main issue they have is multiple database support. Music to my
ears (and especially Alex's :) since it looks like 1.2 will solve most
(all?) of their issues. My main takeaway from this point was that the
best way to please companies like this one will be to ship 1.2 on
time. Which will happen. So we rock.
The next big one was the "startup signal" issue -- they've got lots of
code that needs to run at startup time, and there's no great mechanism
to do that currently. The core devs have talked about this one a *lot*
over the years, and it's obviously a hard one -- for one, there's no
clear definition of what "startup" means -- but a common theme seems
to be that bigger, more complex applications need some way to cleanly
run one-time, expensive startup tasks.
Next, we discussed the difficulty of maintaining large installations
with multiple sites, a plethora of apps, and unspeakable possible
combinations thereof. Django's settings mechanism is simple and easy
to use, but once you have hundreds of sites it starts getting really
tricky to keep things in sync across hundreds of settings files. We
talked at some length about different possibilities for dynamic
settings infrastructure; this would especially come in handy for folks
wanting to build application servers/services like App Engine or the
nascent Toppcloud.
<digress>If you've not yet checked out Toppcloud, do so now. I'll
wait.</digress>
Finally, we ruminated over the difficulties in building rich internet
applications. Sure, writing HTML/CSS/JS/Python/SQL by hand works fine,
but we doesn't really have a good answer for the people who want
something IDE or GUI-ish. Meanwhile, Adobe and Microsoft are putting
all sorts of marketing dollars into Flex/Silverlight, and although
HTML5 can do some amazing things, the lack of tooling is a big danger.
(I've written at more length about this in the past:http://jacobian.org/writing/snakes-on-the-web/#s-rich-web-applications).
Of course, there may not be much us backend folks can do about this
problem -- I'm certainly not enough of a GUI application guy to be
able to even start to think about this problem -- but the lack of an
end-to-end solution will nonetheless put some off from writing web
applications with open source tools.
So there we are. This is very much a brain dump, and I don't really
expect any concrete action to result from it. However, I found some
really interesting stuff there, and I thought I'd share.
Jacob
mrts
2010-01-20 06:46:37 UTC
Permalink
Post by David Cramer
The first three have been huges ones with us. We're just now running
into the settings issue, but would love to see what people can come up
with for solutions (we dont have a good one).
The override_settings() idea that Eric Florenzano describes at
http://djangodose.com/articles/2009/09/handling-development-staging-and-production-enviro/
with some improvements and fixes as described in the following
http://djangodose.com/articles/2009/09/handling-development-staging-and-production-enviro/#comment-16302200
have made my life considerably easier.

Hope the following helps others as well.

# 1.
# settings is a directory, containing host-specific overrides
# and the generic override logic

$ ls settings/
__init__.py override.py host1.py host2.py

# 2.
# __init__.py has the same contents that settings.py
# usually has, except it automatically imports host-specific
# settings in the end

$ tail -2 settings/__init__.py
override.override_settings('projectname', platform.node().split('.')
[0],
sys.modules[__name__])

# 3.
# The generic override logic is as follows

$ cat settings/override.py
from django.utils.importlib import import_module
import warnings

def override_settings(package, name, settings_module):
rel_path = 'settings.' + name
try:
override_module = import_module(rel_path, package=package)
except ImportError:
warnings.warn("Failed to import settings from '%s.%s'" %
(package, rel_path))
else:
for key in dir(override_module):
if key.isupper() and not key.startswith('__'):
setattr(settings_module, key, getattr(override_module,
key))
Mathieu Leduc-Hamel
2010-01-20 19:50:12 UTC
Permalink
Using djangorecipe and zc.buildout offer your exactly that kind of mechanism.

You write the name of settings of choice in your buildout:

[django]
recipe = djangorecipe
version = 1.1
settings = development
eggs = ${eggs:eggs}
wsgi = true
project = project

And after that you can have by example another production's
configuration file like "prod.cfg" overloading your django part and
using another settings name.

In your project dir, you'll have a file named development.py and
another one named prod.py. In these files you can override any
configuration you want.

For deployement, zc.buildout is working pretty fine with django:

http://pypi.python.org/pypi/djangorecipe
Post by mrts
Post by David Cramer
The first three have been huges ones with us. We're just now running
into the settings issue, but would love to see what people can come up
with for solutions (we dont have a good one).
The override_settings() idea that Eric Florenzano describes at
http://djangodose.com/articles/2009/09/handling-development-staging-and-production-enviro/
with some improvements and fixes as described in the following
http://djangodose.com/articles/2009/09/handling-development-staging-and-production-enviro/#comment-16302200
have made my life considerably easier.
Hope the following helps others as well.
# 1.
# settings is a directory, containing host-specific overrides
# and the generic override logic
$ ls settings/
__init__.py   override.py   host1.py   host2.py
# 2.
# __init__.py has the same contents that settings.py
# usually has, except it automatically imports host-specific
# settings in the end
$ tail -2 settings/__init__.py
override.override_settings('projectname', platform.node().split('.')
[0],
       sys.modules[__name__])
# 3.
# The generic override logic is as follows
$ cat settings/override.py
from django.utils.importlib import import_module
import warnings
   rel_path = 'settings.' + name
       override_module = import_module(rel_path, package=package)
       warnings.warn("Failed to import settings from '%s.%s'" %
               (package, rel_path))
               setattr(settings_module, key, getattr(override_module,
key))
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to django-developers+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
aviahlaor
2010-01-20 09:37:38 UTC
Permalink
Startup signal - when using mod_wsgi the wsgi init app can be a good
place to call the startup work, starting threads, setting logging
level etc. It worked for me, but it's probably much simpler than a
complex corporate deployment.
Post by Jacob Kaplan-Moss
Hi folks --
I had a conversation this morning with one of my clients. In the
interests of being a good corporate citizen I'll refrain from
mentioning who other than (a) they're big (Fortune 1000), (b) you've
heard of them, and (c) they're using Django.
Before our chat, they'd invited any engineers internally to bitch
about Django, and they shared some of the pain points they'd come
across. I took some notes; the really good news is that it's a short
list, and we already know about most of it.
Before I share the list, I should mention that I'm not arguing we have
to immediately devote resources to these problems, nor even that we
have to solve them. I'm sharing this just as food for thought, and as
a perspective from a class of developers who mostly don't participate
in our process at all.
The main issue they have is multiple database support. Music to my
ears (and especially Alex's :) since it looks like 1.2 will solve most
(all?) of their issues. My main takeaway from this point was that the
best way to please companies like this one will be to ship 1.2 on
time. Which will happen. So we rock.
The next big one was the "startup signal" issue -- they've got lots of
code that needs to run at startup time, and there's no great mechanism
to do that currently. The core devs have talked about this one a *lot*
over the years, and it's obviously a hard one -- for one, there's no
clear definition of what "startup" means -- but a common theme seems
to be that bigger, more complex applications need some way to cleanly
run one-time, expensive startup tasks.
Next, we discussed the difficulty of maintaining large installations
with multiple sites, a plethora of apps, and unspeakable possible
combinations thereof. Django's settings mechanism is simple and easy
to use, but once you have hundreds of sites it starts getting really
tricky to keep things in sync across hundreds of settings files. We
talked at some length about different possibilities for dynamic
settings infrastructure; this would especially come in handy for folks
wanting to build application servers/services like App Engine or the
nascent Toppcloud.
<digress>If you've not yet checked out Toppcloud, do so now. I'll
wait.</digress>
Finally, we ruminated over the difficulties in building rich internet
applications. Sure, writing HTML/CSS/JS/Python/SQL by hand works fine,
but we doesn't really have a good answer for the people who want
something IDE or GUI-ish. Meanwhile, Adobe and Microsoft are putting
all sorts of marketing dollars into Flex/Silverlight, and although
HTML5 can do some amazing things, the lack of tooling is a big danger.
(I've written at more length about this in the past:http://jacobian.org/writing/snakes-on-the-web/#s-rich-web-applications).
Of course, there may not be much us backend folks can do about this
problem -- I'm certainly not enough of a GUI application guy to be
able to even start to think about this problem -- but the lack of an
end-to-end solution will nonetheless put some off from writing web
applications with open source tools.
So there we are. This is very much a brain dump, and I don't really
expect any concrete action to result from it. However, I found some
really interesting stuff there, and I thought I'd share.
Jacob
Vinay Sajip
2010-01-20 15:37:33 UTC
Permalink
Post by Jacob Kaplan-Moss
The next big one was the "startup signal" issue -- they've got lots of
code that needs to run at startup time, and there's no great mechanism
to do that currently. The core devs have talked about this one a *lot*
over the years, and it's obviously a hard one -- for one, there's no
clear definition of what "startup" means -- but a common theme seems
to be that bigger, more complex applications need some way to cleanly
run one-time, expensive startup tasks.
I've got a mechanism which I described in these posts:

http://groups.google.com/group/django-developers/msg/2b94fa2017b07e67

and

http://groups.google.com/group/django-developers/msg/7f14fc0d9ee92b43

The thread refers to logging but the approach works for any start up
code;
I've provided three points where code can be hooked, which I've called
"bootstrap" (e.g. to initialize logging), "pre-model" (before app
loading, say to
install class_prepared listeners) and "post-model" (after app loading,
to do
any work which needs apps to be initialized and models to be
available).

I would welcome some feedback ... Launchpad branch is here:

https://code.launchpad.net/~vinay-sajip/django/logging

An example settings file which uses the functionality in the branch is
in the
second of the two posts mentioned above.

Regards,

Vinay Sajip
sago
2010-01-20 17:15:39 UTC
Permalink
I also work with several fortune 500 clients (though not specifically
as a Django consultant, Django is a preferred tool) and the issues
your client mentions, Jacob, are very similar to those I deal with.

Some other notes from the concerns of multinational clients:

I've had one very long and complex issue with a major client over
legacy databases with Composite Primary Keys (and other composite keys
more generally), an issue which has also come up in other contexts.
One of my smaller clients switched to a strange bastardization of
Django and SQLAlchemy because this was an issue. I understand the
history of #373 and why it is hard - just for disclosure.

A lack of any mechanism for developers to understand what third party
apps are reliable, what to avoid, known problems, and so on. This has
been a particular complaint since the Django ethos is (rightly) that
the core is the core, and other functionality should be packaged into
third party apps. App authors do a great job of being really generous
with their work, but it is very hard for stuff outside Django's core
to get any adoption. The processes involved in adopting a technology
can be complex and involve lots of stakeholders, legal and technical
due diligence. It is hard enough ticking those boxes to get Django
used, it simply isn't worth doing the same for all the other bits.
There's a lot of wheel reinventing going on, in my experience. A Django
+ project might be useful to give the rock-star third-party apps a
fighting chance at being used. Pinax is somewhat close, but has a very
different purpose and small, agile, company ethos.

Startup and Settings are the big killers though, head and shoulders
above the previous issues. I've nothing much to add to your comments
other than to say that some of the Django deployments I know of are
highly heterogeneous, with various servers running different SOA
elements, some Django some not. The configuration issues are
formidable and involve all kinds of dependencies beyond Django (or
even Python). I struggle to see a way that Django could solve those
issues, but allowing more flexibility and avoiding tying configuration
to an executable module (i.e. settings.py) would be useful. Like the
previous posters, I've also had to roll various custom solutions to
this problem. Is suspect at least some of the solution will always be
custom, though it might be worth making such solutions easier to
create.

It would also be useful for everyone to scream more loudly that monkey-
patching ANYTHING is a very bad idea. Its getting better now (as some
of the major things it was solving are solved properly), but it was a
woefully overused technique for a while that has meant a lot of
deployment pain! That's less about Django and more about evangelizing
bad development practice under the guise of Python-Fu.

Ian.
Luke Plant
2010-01-20 18:31:24 UTC
Permalink
Post by sago
Startup and Settings are the big killers though, head and shoulders
above the previous issues. I've nothing much to add to your
comments other than to say that some of the Django deployments I
know of are highly heterogeneous, with various servers running
different SOA elements, some Django some not. The configuration
issues are formidable and involve all kinds of dependencies beyond
Django (or even Python). I struggle to see a way that Django could
solve those issues, but allowing more flexibility and avoiding
tying configuration to an executable module (i.e. settings.py)
would be useful.
I don't understand how avoiding the settings.py mechanism will produce
*more* flexibility. With an executable module storing settings, you
can implement whatever mechanism you want for settings:

## settings.py ##

import sys
from acme import network_config

network_config.load_django_settings_into_module(sys.modules[__name__])

## end.

Granted, code that does the above might need to be written, but the
point of settings.py is that you *can* write it, without forcing
anyone to use that system.

Luke
--
"Procrastination: Hard work often pays off after time, but laziness
always pays off now." (despair.com)

Luke Plant || http://lukeplant.me.uk/
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to django-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Jacob Kaplan-Moss
2010-01-20 18:54:08 UTC
Permalink
Post by Luke Plant
I don't understand how avoiding the settings.py mechanism will produce
*more* flexibility.
The problem -- at least as I see it -- is that of a intertwingulment
of "application" settings with "ops" settings.

As organizations get larger, roles stratify more. Thus you end up with
a pretty big gulf between the ops team and the application developers.
In these environments, application developers probably shouldn't be
messing with things like the `DATABASE_*` settings -- in some
organizations its even a breach of policy for the developers to *know*
the database password!

OTOH, ops types don't know what the heck to do with `INSTALLED_APPS`...

And then there's middleware: some middleware (caching, etags) might
need to be controlled by ops; some by apps.

Obviously you can *do* this with Django, and I'd argue it's not
particularly hard. Still, it's something that bigger orgs have to
figure out themselves, and that can be a barrier to adoption.

The enterprisey world solves this with standards like WAR and servers
like Glassfish: if I'm an application developer, all I need to do to
deploy is somehow produce a WAR. And if I'm a ops person, I just take
that WAR and slap it into Glassfish. It's rarely that easy in
practice, of course, but the execs making the tech decisions don't
know that.

Jacob
Andy McKay
2010-01-20 19:52:09 UTC
Permalink
Post by Jacob Kaplan-Moss
Post by Luke Plant
I don't understand how avoiding the settings.py mechanism will produce
*more* flexibility.
The problem -- at least as I see it -- is that of a intertwingulment
of "application" settings with "ops" settings.
+1
--
Andy McKay, @clearwind
http://clearwind.ca/djangoski
Honza Král
2010-01-21 01:59:49 UTC
Permalink
Post by Jacob Kaplan-Moss
Post by Luke Plant
I don't understand how avoiding the settings.py mechanism will produce
*more* flexibility.
The problem -- at least as I see it -- is that of a intertwingulment
of "application" settings with "ops" settings.
As organizations get larger, roles stratify more. Thus you end up with
a pretty big gulf between the ops team and the application developers.
In these environments, application developers probably shouldn't be
messing with things like the `DATABASE_*` settings -- in some
organizations its even a breach of policy for the developers to *know*
the database password!
OTOH, ops types don't know what the heck to do with `INSTALLED_APPS`...
And then there's middleware: some middleware (caching, etags) might
need to be controlled by ops; some by apps.
Obviously you can *do* this with Django, and I'd argue it's not
particularly hard. Still, it's something that bigger orgs have to
figure out themselves.......
That's a problem that we have. We solve it very simply by splitting
settings.py into a package and dividing these options. We also provide
some machine wide configuration in /etc/ that won't get overwritten
during update of the project - that's where ops guys store their
passwords and everything.

the mechanism is shortly described here:
http://ella.github.com/tutorial/4-deployment.html#settings-py
project template:
http://github.com/ella/django-base-project/tree/master/djangobaseproject/

I think this can be solved by documenting different approaches (like
http://code.djangoproject.com/wiki/SplitSettings ) or maybe even
picking one and recommending that.





Another problem that we have is error reporting. If your project blows
up and starts generating hundreds of error messages per second, you
really don't want then on your mailserver or mail client. It is also
solvable with django's current tool set, but everybody must also
figure it out on their own.

We are currently working on some simple mechanism to handle this
(offload the exceptions into a queue, group the same errors and store
it in some DB or just log it), we will share the code as soon as it's
finished.
Andy McKay
2010-01-21 02:39:38 UTC
Permalink
Post by Honza Král
Another problem that we have is error reporting. If your project blows
up and starts generating hundreds of error messages per second, you
really don't want then on your mailserver or mail client. It is also
solvable with django's current tool set, but everybody must also
figure it out on their own.
There's quite a few other solutions for that. Eg: writing to Apache (if you use it logs, writing to sys log etc) I think the new logging framework addition will help here. Or there are services like Arecibo: http://areciboapp.com/listener/docs/django/ (other services are available) to do that for you.
--
Andy McKay, @clearwind
http://clearwind.ca/djangoski
Russell Keith-Magee
2010-01-21 07:55:18 UTC
Permalink
Post by Jacob Kaplan-Moss
Post by Luke Plant
I don't understand how avoiding the settings.py mechanism will produce
*more* flexibility.
The problem -- at least as I see it -- is that of a intertwingulment
of "application" settings with "ops" settings.
I'm in full agreement that this is a problem.

However, isn't the solution just a matter of adequately documenting
the options that are already available? We could certainly improve the
documentation for complex deployments - e.g., pointing out that:
* settings.py can import base_settings.py
* DJANGO_SETTINGS_MODULE lets you call your settings.py whatever you want
* documenting which settings are operations, and which are application

I'm not convinced that we have anything particularly technical to address here.
Post by Jacob Kaplan-Moss
And then there's middleware: some middleware (caching, etags) might
need to be controlled by ops; some by apps.
Obviously you can *do* this with Django, and I'd argue it's not
particularly hard. Still, it's something that bigger orgs have to
figure out themselves, and that can be a barrier to adoption.
It strikes me that this is just one example of a whole class of "best
practice" stuff that we should document, but don't at present - like
project layout, automated deployment, continuous integration,
configuration management, etc.

We've historically avoided having documentation on how to use Django
with external tools (except when we have to, as in the case of
mod_wsgi). A big part of the solution to the problem you've described
is to establish a vetted best practice, and documenting that practice
on our website as a community resource.

The rest of the solution falls to the toolchain outside of Django
itself. It still needs to be solved, but it isn't a Django problem
specifically. One option here might be to take the Apache model, and
use the DSF to incubate tools that have benefits to Django. However,
that's part of a much bigger discussion.

Yours,
Russ Magee %-)
gareth rushgrove
2010-01-21 09:13:56 UTC
Permalink
Post by Russell Keith-Magee
It strikes me that this is just one example of a whole class of "best
practice" stuff that we should document, but don't at present - like
project layout, automated deployment, continuous integration,
configuration management, etc.
We've historically avoided having documentation on how to use Django
with external tools (except when we have to, as in the case of
mod_wsgi). A big part of the solution to the problem you've described
is to establish a vetted best practice, and documenting that practice
on our website as a community resource.
I'd certainly be interested in contributing to this effort. I have
lots of this information scattered about on my blog, internal company
notes and presentations from DJUGL and the likes. I've avoided trying
to get it on the wiki for the reasons you describe.

What is the best way of going about documenting this so it can be
vetted by the community and enshrined as somewhat official best
practice - without the work in progress being used before it's been
stamped in someway?
Post by Russell Keith-Magee
The rest of the solution falls to the toolchain outside of Django
itself. It still needs to be solved, but it isn't a Django problem
specifically. One option here might be to take the Apache model, and
use the DSF to incubate tools that have benefits to Django. However,
that's part of a much bigger discussion.
Yours,
Russ Magee %-)
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
--
Gareth Rushgrove

Web Geek


morethanseven.net
garethrushgrove.com
gareth rushgrove
2010-01-21 09:09:02 UTC
Permalink
Post by Jacob Kaplan-Moss
Post by Luke Plant
I don't understand how avoiding the settings.py mechanism will produce
*more* flexibility.
The problem -- at least as I see it -- is that of a intertwingulment
of "application" settings with "ops" settings.
As organizations get larger, roles stratify more. Thus you end up with
a pretty big gulf between the ops team and the application developers.
In these environments, application developers probably shouldn't be
messing with things like the `DATABASE_*` settings -- in some
organizations its even a breach of policy for the developers to *know*
the database password!
In the past I've used code like the following in settings files:

# we're going to allow overriding of settings via a yml file.
# This makes live nicer for anyone managing the box
# and means settings can be overloaded without redeploying the site
SETTINGS_OVERIDE = "/etc/$(project).yml"
try:
file = open(SETTINGS_OVERIDE)
for key, value in yaml.load(file).items():
globals()[key]=value
except:
# we don't always have the file around or need the setting
# defined so best to be quite if things go wrong
pass

You can also make this more limited, by allowing only the overriding
certain settings. You can then use tools like puppet on the ops side
to push config files in to /etc (or wherever suites you system).

Example here http://morethanseven.net/2009/02/15/let-you-sys-admin-override-your-django-settings/

This pattern is also used in my project templates paster stuff:
http://github.com/garethr/django-project-templates and
http://pypi.python.org/pypi/django-project-templates/
Post by Jacob Kaplan-Moss
OTOH, ops types don't know what the heck to do with `INSTALLED_APPS`...
And then there's middleware: some middleware (caching, etags) might
need to be controlled by ops; some by apps.
Obviously you can *do* this with Django, and I'd argue it's not
particularly hard. Still, it's something that bigger orgs have to
figure out themselves, and that can be a barrier to adoption.
The enterprisey world solves this with standards like WAR and servers
like Glassfish: if I'm an application developer, all I need to do to
deploy is somehow produce a WAR. And if I'm a ops person, I just take
that WAR and slap it into Glassfish. It's rarely that easy in
practice, of course, but the execs making the tech decisions don't
know that.
Jacob
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
--
Gareth Rushgrove

Web Geek


morethanseven.net
garethrushgrove.com
sago
2010-01-20 20:12:23 UTC
Permalink
Post by Luke Plant
I don't understand how avoiding the settings.py mechanism will produce
*more* flexibility.
Bad choice of words on my part, sorry. Of course code is always more
flexible than static data.

The two key bits of required 'flexibility' I didn't have out of the
box (and were difficult to hack and get working) are:
1. App level settings that are mutually exclusive (one app needs
settings different from another - this happens a lot).
2. Setting definitions that aren't tied in timing and scope terms to
Python's import process (I've seen this interact badly with start-up
code, for example).
And it would be cool to have a level of indirection so that settings
could be guaranteed to be dynamic (some are, some aren't, those that
are often need monkey-patch style hacks to change). And tools for
debugging where settings for a failed request are defined. And a
security layer around some settings. And fields and fields of other
ponies.

Seriously though, I don't think the settings.py approach is broken. It
makes sense, makes getting from nothing to runnable code a breeze, is
logical for small deployments, and has bags of flexibility for most
use-cases. I'm happy adjusting it in my projects. I've never deployed
or seen deployed a system that used settings.py in anything like its
default version, but the replacements aren't standard enough to be an
alternative default; every client has different technical, management
and internal politics constraints.

My comments were meant in the spirit of Jacob's. Just FYIs for a
particular niche of users.

I should also say that most of the developers I work with do love
working with Django. So their criticisms have to be taken against a
background of strong positivity.
mtnpaul
2010-01-20 19:09:32 UTC
Permalink
Post by sago
A lack of any mechanism for developers to understand what third party
apps are reliable, what to avoid, known problems, and so on. This has
been a particular complaint since the Django ethos is (rightly) that
the core is the core, and other functionality should be packaged into
third party apps. App authors do a great job of being really generous
with their work, but it is very hard for stuff outside Django's core
to get any adoption. The processes involved in adopting a technology
can be complex and involve lots of stakeholders, legal and technical
due diligence. It is hard enough ticking those boxes to get Django
used, it simply isn't worth doing the same for all the other bits.
There's a lot of wheel reinventing going on, in my experience. A Django
+ project might be useful to give the rock-star third-party apps a
fighting chance at being used. Pinax is somewhat close, but has a very
different purpose and small, agile, company ethos.
While not perfect I find how the http://plugins.jquery.com/ site
works. It gives me more confidence in picking a plugin to use for a
particular project to see how other users have rated an app. The demos
and examples of the plugins is also very nice. Something similar for
Django apps would be great.
PauloS
2010-01-22 13:53:28 UTC
Permalink
Post by sago
I've had one very long and complex issue with a major client over
legacy databases with Composite Primary Keys (and other composite keys
more generally), an issue which has also come up in other contexts.
One of my smaller clients switched to a strange bastardization of
Django and SQLAlchemy because this was an issue. I understand the
history of #373 and why it is hard - just for disclosure.
Composite primary keys is a major issue over here and we also ended up
with a bastard of SQLAlchemy and Django.

--
Paulo
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to django-developers+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Luke Plant
2010-01-21 14:58:24 UTC
Permalink
Post by Jacob Kaplan-Moss
So there we are. This is very much a brain dump, and I don't really
expect any concrete action to result from it. However, I found some
really interesting stuff there, and I thought I'd share.
Thanks for your clarification elsewhere in this thread.

One question I have (for Jacob and anyone else with experience):

I would have guessed that a big issue with Django from an enterprise
perspective is its use of 'singletons'. Though we don't use that
terminology, we have dozens of them in the code - every other setting
we have implies the existence of a singleton, since the setting can
have only one value. For example, EMAIL_BACKEND means that we have
one email component singleton which is used everywhere. If you wanted
one set of views to use a different email backend, you are out of
luck.

The 'admin' app was the first major component to allow multiple
instances, but there are various other apps which might need this
change in theory.

How much is this an issue in practice?

I imagine that for most sub-systems, most people actually want
singletons, but for the cases you don't, it is a big pain. Personally
I think that, while there has been an unhealthy proliferation of
settings which we need to trim, Django has hit the sweet spot between
easy set up and configuration on the one hand, and flexibility on the
other. The alternative - turning everything into components with
dependency injection everywhere - could be horrific for our sweet
spot.

One solution to this problem would be to have multiple Django
instances hosting different parts of your URL space, allowing them to
have multiple settings files. This has obvious limitations, but from
the perspective of a large project, perhaps it's not too bad - I
imagine that large sites are already used to carving up the URL space
to be served by completely different technologies in some cases, at
least for hosts running LAMP or similar.

Regards,

Luke
--
"Procrastination: Hard work often pays off after time, but laziness
always pays off now." (despair.com)

Luke Plant || http://lukeplant.me.uk/
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to django-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Peter Herndon
2010-01-21 16:05:30 UTC
Permalink
Post by Luke Plant
Post by Jacob Kaplan-Moss
So there we are. This is very much a brain dump, and I don't really
expect any concrete action to result from it. However, I found some
really interesting stuff there, and I thought I'd share.
Thanks for your clarification elsewhere in this thread.
I would have guessed that a big issue with Django from an enterprise
perspective is its use of 'singletons'. Though we don't use that
terminology, we have dozens of them in the code - every other setting
we have implies the existence of a singleton, since the setting can
have only one value. For example, EMAIL_BACKEND means that we have
one email component singleton which is used everywhere. If you wanted
one set of views to use a different email backend, you are out of
luck.
The 'admin' app was the first major component to allow multiple
instances, but there are various other apps which might need this
change in theory.
How much is this an issue in practice?
Speaking as a web developer using Django in a *very* enterprisey corporation, I will say this singleton issue has been a problem for me. As a self-inflicted example, I wrote an LDAP backend that looks up the necessary settings from settings.py. I need to rewrite it to store the settings in the db, because I need to be able to query multiple LDAP sources. I run into similar issues with lots of different apps. So yes, singletons are an issue for me.

The difference between ops settings vs. app settings has also been a problem. The lack of established, vetted best practices (for deployment, layout, everything Russ mentioned) also has bitten me regularly. In part because I wind up spending a lot of time doing the research to figure out what I need to do, in part because the state of the art moves forward at a rapid pace. But, to put it in perspective, I'm the only person who even *can* deploy a Django app at my place of work. Which is not great, but I'm also a Linux sys-admin, so it is at least in my job description. But none of my fellow SAs have any clue what to do to get stuff working on one of our corporate RHEL 5 boxen, and no inclination to learn.

Anyway, I'm glad to see this discussion taking place. Django indeed has a sweet spot, and it isn't really for enterprise installations -- though it is *very* good in that sweet spot. There are places where the early design decisions to favor convention over configuration impose a lack of flexibility in favor of simplicity. A definite trade-off. It makes it harder to use in places where flexibility and integration are required.

Here's a big question, though: given that satisfying the enterprise crowd, of which I am an occasional part, will require additional "stuff" that may bring along with it additional complexity, making it potentially harder to get started for newbies, is that a direction that the project should head? Is it worth satisfying the enterprise crowd to increase adoption, at the cost of turning into the kind of configuration hell that J2EE servers exemplify? Or, more positively, is it possible to add the required enterprisey bits while still maintaining a certain level of simplicity? Note that Django is already headed towards greater complexity (multi-db, email backends), due to requirements from the current community.

My two cents, appropriately adjusted for the current economic situation.

---Peter Herndon
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to django-developers+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
sago
2010-01-21 16:24:16 UTC
Permalink
Post by Luke Plant
I would have guessed that a big issue with Django from an enterprise
perspective is its use of 'singletons'.
How much is this an issue in practice?  
Yes, that's been my experience. For complex deployments with tens or
hundreds of apps, it is often the case that it is absolutely essential
to have settings on a per-app basis. Unfortunately those settings very
often clash.
Post by Luke Plant
It strikes me that this is just one example of a whole class of "best
practice" stuff that we should document, but don't at present - like
project layout, automated deployment, continuous integration,
configuration management, etc.
It would help smaller companies I think, folks who need to deploy a
couple of SOA nodes or a site with ten or twenty apps across a handful
of servers. When it comes to huge deployments, it is so diverse. Best
practice for Google isn't best practice for wisconsinhairstylists.com

Case studies are helpful if folks can share them. But its a political
minefield (or unhelpful pontification, depending on the care you take)
to get into the business of canonizing deployment structures!
Jacob Kaplan-Moss
2010-01-21 16:49:06 UTC
Permalink
Post by sago
Case studies are helpful if folks can share them.
Just a quick note that the DSF is working on gathering case studies;
getting folks who have permission to share is indeed a tricky issue.

Not much is set up yet, but I, at least, would be *thrilled* to help
companies using Django put together case studies that could be shared
publicly. Until we've got something more formal in place, consider it
a standing offer -- drop me a line if you'd like to share a success
story.

Jacob
Antoni Aloy
2010-01-21 17:26:11 UTC
Permalink
Hi!

My previous experience was from Java J2EE world and as Jacob said we
adopted Django because it really shines solving some complexity
problems associated to J2EE world as MVC patter, templates, etc., but
I agree that a common way to configure external applications would be
nice.

I'll also have been searching a good RIA to connect with Django. One
of the frameworks I like best is Qooxdoo http://qooxdoo.org/, perhaps
the best javascript opensource RIA frameworks I have seen. In order to
connect with Qooxdoo just Json-RCP or xml-rcp is needed. Perhaps we'll
need an special support for this in Django to mark a view as rcp as
some external projects do as RCP4Django or other. I'd like to see
Qooxdoo as the default Django RIA (not the default javascript
framework) as Qooxdoo people uses also Python and the framework fits
quite well in what I think is an enterprise RIA framework.

I know that this could be against the Django policy of not to bless a
javascript framework, but in my opinion if we want to shine at RIA you
have to choose.

We have created also some HTTP-XML based web services in Django in a
fraction of time we needed to create them in Java. For that we are
using Django without ORM, but we have found a great help to allow
documenting the web service and create the test pages.

Anyway, I'm glad to see Fortune companies are interested in Django :)
--
Antoni Aloy López
Blog: http://trespams.com
Site: http://apsl.net
Jacob Kaplan-Moss
2010-01-21 16:47:13 UTC
Permalink
Post by Luke Plant
I would have guessed that a big issue with Django from an enterprise
perspective is its use of 'singletons'.
[snip]
Post by Luke Plant
How much is this an issue in practice?
Other than databases, it hasn't been much of an issue with most of the
BigCos I've been involved with. That's because...
Post by Luke Plant
One solution to this problem would be to have multiple Django
instances hosting different parts of your URL space, allowing them to
have multiple settings files.
Bingo.

All of the larger Django installations I've worked with haven't been
just one project -- they've been dozens (hundreds in one case) of
little bits, often cobbled together by different teams and sometimes
different companies (in-house vs. contractors vs. off-the-shelf
products).

Django's URL/view decoupling looks *really* nice in these cases --
Django apps, properly written, don't care *where* in the URL space
they live -- so this means that Django fits in well in these
heterogenous environments. And I've yet to encounter a larger company
that's anything but wildly heterogeneous.

Peter's question below about enterprisy complexity and the dangers of
becoming J2EE are good ones; I'm glad they're part of the discussion.
It's difficult to know where to draw the line between simplicity and
flexibility.

Right now we have exactly 90 documented settings, some of which point
at classes or interfaces that themselves have more complexity. To me
this already sounds way too high, but it's nothing compared to the
configuration options available in other environments -- PHP has about
600 php.ini settings! -- so there's obviously a matter of taste here.

It's also important to keep in mind that part of the reason Django's
had (some) success in corporate environments is *because* of the
simplicity, not despite it. I've heard, over and over again, that
teams that have switched to Django have done so at least in part
because of the simplicity and ease-of-use.

So we should be careful about setting up a false dichotomy where us
hippie open source freaks only value simplicity and build toys and
those square suit types who love FactoryManagerFactories because CIO
magazine tells them to. The real world, as always, is a lot more
complex.

Jacob
c***@public.gmane.org
2010-01-21 17:15:49 UTC
Permalink
I agree that managing settings gets to be a bit difficult in many
environments - even non Fortune 1000 environments. One of the things
we've played with in Satchmo is a database backed solution that allows
you to define reasonable defaults - even override-able in the settings
- but allow people to edit through the admin interface.

Bruce has been working to split it out from Satchmo here -
http://bitbucket.org/bkroeze/django-livesettings/overview/

I'm not saying this is the end all be all solution but it's something
we've been using for a bit so it should be useful in informing the
direction the community wants to take. I also have some ideas for how
it should be improved or tweaked but in my opinion it's a nice start
and it would be good to see what people think about using it in other
apps. It's also consistent with the Django philosophy of building it
outside of core and seeing if it reaches critical mass for the
community.

-Chris
Andy McKay
2010-01-21 17:45:49 UTC
Permalink
Post by c***@public.gmane.org
I agree that managing settings gets to be a bit difficult in many
environments - even non Fortune 1000 environments. One of the things
we've played with in Satchmo is a database backed solution that allows
you to define reasonable defaults - even override-able in the settings
- but allow people to edit through the admin interface.
I've seen that route before and I have to say that I would be strongly against that. The only saving grace could be multi-db which could allow you to have a specific settings database that's separate from the core.

Having data and settings intermingled in the same area just causes too many problems. Further, it detracts from the ability to run the project without the database around. In the scenario of a larger environment with more delineation of roles, it may not be possible.
--
Andy McKay, @clearwind
http://clearwind.ca/djangoski
stuff4ash
2010-02-10 03:44:08 UTC
Permalink
Post by c***@public.gmane.org
I agree that managing settings gets to be a bit difficult in many
environments - even non Fortune 1000 environments.
Its also an issue for us that write and use portable apps. Luckily a
way for apps to provide defaults could easily be implemented in a
admin.autodisover sort of way.
Somebody already implemented something similar to what i wanted to
write today: http://code.djangoproject.com/ticket/1282

For the rest of the "probelms", as someone who is constantly
integrating heterogenous systems with django, I think the solution
might be "outside" of django.
Some problems are sometimes dealt easier in python space. Just as a
deliverance similar/friendly solution might be a better solution than
making a django admin skin "engine", a more generic configuration app
might be better for advanced uses.
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to django-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Luc Saffre
2010-02-11 00:05:50 UTC
Permalink
Post by Jacob Kaplan-Moss
Finally, we ruminated over the difficulties in building rich internet
applications. Sure, writing HTML/CSS/JS/Python/SQL by hand works fine,
but we doesn't really have a good answer for the people who want
something IDE or GUI-ish. Meanwhile, Adobe and Microsoft are putting
all sorts of marketing dollars into Flex/Silverlight, and although
HTML5 can do some amazing things, the lack of tooling is a big danger.
http://jacobian.org/writing/snakes-on-the-web/#s-rich-web-applications).
It's a bit early to speak about Lino, but Jacob's post incited me to
write a kind of preview for Django developers:

http://code.google.com/p/lino/wiki/LinoAndDjango

Encouraging words are welcome now, but wait a few months before you
seriously investigate this project.

Luc
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-developers-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to django-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Continue reading on narkive:
Loading...