Hi Samuel,
You'll notice if you look carefully that I avoid saying "async" almost
anywhere in the Channels announcements or documentation, and when I do,
it's qualified very carefully. Channels makes Django event-driven, but it
does not provide full "in-process" async like Twisted, asyncio, or
something like Pulsar.
Channels is actually kind of related to the actor model in some ways, but
it's simpler; imagine it as a pool of identical actors that are all
stateless, apart from the interface servers, which have the incoming socket
and thus the state.
The key thing is that Channels does nothing to make Django asynchronous.
All the Django code will remain synchronous (the ORM, templating, etc.),
and run in synchronous processes that just process one message after the
next. The only part of the system that will require an async framework is
the interface servers, as to be anywhere near efficient they need to serve
more than one socket at once; right now we're using Twisted and asyncio for
those, purely as they're incredibly mature and well-supported by the
community, but you could swap out anything there that achieved the same
goal (greenlets, for example, though I personally dislike their
implicitness).
Channels essentially takes the worker-prefork model that's been used in
webservers for years and applies it to a lower-level message concept rather
than HTTP requests (http requests are just one kind of message). This not
only has the advantage of a lot of previous work and research into
architecting and scaling these kinds of systems, it also means the actual
amount of work in core Django isn't crazy; it's more about providing nice
places to wrap up the worker loop and nice utilities around authentication
and session management.
If you read through the docs there's a bit more about this, but basically,
the fundamental design goal here is that Channels be no harder to use than
normal Django; indeed, it's almost exactly the same view concept, but
instead of a request you get a message, and the newer developers I've
interviewed about it seem to be comfortable with the extension of the
current abstraction straight away. I don't ever want people to have to
write asynchronous code if they don't want to; it's currently so easy to
shoot yourself in the foot in almost any Python async solution that I don't
think that makes for a good developer experience.
The "complexity" of the message backends you describe is because they're
network-transparent; there is indeed a simple, in-process one that uses
threads too, but the whole point of this design is that you run Django over
multiple processes on many machines, and in order to get something like
that you need a place to funnel messages through (another potential backend
design could use direct connection and process discovery, perhaps, or an
Erlang-like process spawn-and-track design with an overarching controller
process with access to many pre-selected machines).
The idea of making an asynchronous Django is tangential to making Channels;
this is more about changing the design abstraction of Django so we can fit
the concept in, and then providing a neat solution that drops into the gap.
There's nothing stopping someone making the worker processes for Channels
async-aware as well, so they can run multiple message consumers at once,
but that's such a complex and risk-fraught task that I don't think it's
worth it when we can get 90% of the benefits with a much simpler and
easier-to-design-and-understand system. If you want to see an example of
actual async Django, see Amber's talk from Django Under the Hood this year,
where she managed to get it working decently well under Twisted.
I expect Channels to smooth out Django's performance mostly as a
side-effect of the worker model providing demand-based load balancing by
design, but I doubt it will be a huge increase until people start using the
primitives it provides to offload tasks to other workers.
I hope that somewhat long-winded explanation helps - if not, let me know
what's still unclear and I'll try to tackle it. Clearing up what we're
trying to do and communicating it is important to me; I want to make sure
everyone knows so we can get good feedback on this stuff. For what it's
worth, I did consider the actor model along with several other kinds of
concurrency approaches (I've been prototyping bits of this for years); this
approach ended up being the nicest design and the one I have the most
confidence in that we can scale and shard to large work volumes.
Andrew
Post by Samuel BishopI'm uncertain how popular the suggestion will be but ... are "channels"
the right solution to our async/concurrency problems? ( I'm all for
django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the
explanations I'm reading as I follow along with the progress of 'Django
Channels", just feels "less good" than solving this problem via Actor Model
Concurrency (championed by Erlang, Elixir, Stackless Python, etc).
There is an excellent actor model concurrency library for python that
already supports 'async django'. "Pulsar" (
http://pythonhosted.org/pulsar/index.html) and the documentation here
http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways
to use actor model concurrency to attack the issue of using Django with
asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels
sounds very similar to what they provide to work around the limitations of
django's blocking request processing cycle other parts sound much more
complicated, for instance the need for various back ends
in-memory/redis/etc.
I know Pulsar is currently Python 3.4 and newer only, but in the past it
supported 2.7, so anything we can learn from their concurrency approach can
work on Python 2.7 to maintain our backwards compatibility going forward.
You can see an example of Pulsar using python 2 if you look on the
benchmark page for their python powered 'redis clone' -
https://gist.github.com/lsbardel/8068579 - One of the benchmarks is using
pypy so the code definitely worked in a python 2.7 environment.
Would the work to integrate channels as part of Django improve the ability
to run Django in a completely async mode and enhance its performance when
run under Pulsar? or would it just be something that needed to be turned
off if we wanted to use something like Pulsar to power our "async django".
Is there anything to be gained from using a more actor driven concurrency
model or adopting any of the other methods used in their 'pulse' django app?
I suppose I'm just trying to work out how to compare using Django-Channels
with using Pulse from the Pulsar project.
On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
Post by Markus HoltermannPost by Shai BergerIf I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
I don't agree; I think it's more "do you have enough workers to handle
this". Channels makes all workers serve all requests, so if you're doing
e.g. video encoding you can't segregate it from request handling. However,
you could have multiple channel backends and thus multiple worker pools to
achieve this.
Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upsqkL9_qN6NfO7PB9xjHMaPrjL9whwpnmPo3Ru4ZbO0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.