Discussion:
MOSS Award to Django
Andrew Godwin
2015-12-11 18:18:27 UTC
Permalink
Hi everyone,

For those who haven't seen, Mozilla has awarded $150,000 to Django for work
on Channels and request/response improvements lifted from REST Framework.
More in the blog post here:
https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/

I'll be coordinating this effort for the most part, but we're still working
out who to fund and the roadmap for the project, as well as work out how we
can pay people for their work on a different scale to the Django
Fellowship, so it might take a bit of time!

I'll be back on here with some questions for people to discuss/answer about
the channels design at some point soon, but a lot of the basic design is
already up at http://channels.readthedocs.org, so take a read over that if
you're interested.

What I can say is that my intention is to both bake Channels into the next
major release of Django, as well as hopefully release a pluggable app
version that will run on 1.8 and 1.9 - I have some plans around how to do
that effectively, but they involve hitherto unforged paths around Django
and how we package dependencies, so I can't say we'll get there 100%.

Andrew
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upjXf2LLk2gGA9mzdPQn4DbWehSHNFWhF-GkEk4q54gHA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Asif Saifuddin
2015-12-12 07:18:14 UTC
Permalink
Awesome !!!! awesome !!! awesome!!! :D :D
Post by Andrew Godwin
Hi everyone,
For those who haven't seen, Mozilla has awarded $150,000 to Django for
work on Channels and request/response improvements lifted from REST
https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
I'll be coordinating this effort for the most part, but we're still
working out who to fund and the roadmap for the project, as well as work
out how we can pay people for their work on a different scale to the Django
Fellowship, so it might take a bit of time!
I'll be back on here with some questions for people to discuss/answer
about the channels design at some point soon, but a lot of the basic design
is already up at http://channels.readthedocs.org, so take a read over
that if you're interested.
What I can say is that my intention is to both bake Channels into the next
major release of Django, as well as hopefully release a pluggable app
version that will run on 1.8 and 1.9 - I have some plans around how to do
that effectively, but they involve hitherto unforged paths around Django
and how we package dependencies, so I can't say we'll get there 100%.
Andrew
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bbeea75a-1026-42f1-9cb4-56be0c8a08b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Ibrahim K
2015-12-12 07:22:16 UTC
Permalink
great!!!
Post by Asif Saifuddin
Awesome !!!! awesome !!! awesome!!! :D :D
Post by Andrew Godwin
Hi everyone,
For those who haven't seen, Mozilla has awarded $150,000 to Django for
work on Channels and request/response improvements lifted from REST
https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
I'll be coordinating this effort for the most part, but we're still
working out who to fund and the roadmap for the project, as well as work
out how we can pay people for their work on a different scale to the Django
Fellowship, so it might take a bit of time!
I'll be back on here with some questions for people to discuss/answer
about the channels design at some point soon, but a lot of the basic design
is already up at http://channels.readthedocs.org, so take a read over
that if you're interested.
What I can say is that my intention is to both bake Channels into the
next major release of Django, as well as hopefully release a pluggable app
version that will run on 1.8 and 1.9 - I have some plans around how to do
that effectively, but they involve hitherto unforged paths around Django
and how we package dependencies, so I can't say we'll get there 100%.
Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/bbeea75a-1026-42f1-9cb4-56be0c8a08b2%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/bbeea75a-1026-42f1-9cb4-56be0c8a08b2%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Regards,
Ibrahim.k
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAAf7Lr7OSpwdi8-DFHvk6Go%2BO_7UGUrE%3DLTJL-D0DQM3Edz0ZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Daniel Chimeno
2015-12-12 01:56:05 UTC
Permalink
Hi,
Just saying congrats to people involved and the community. Great step for
Django!

El viernes, 11 de diciembre de 2015, 19:19:00 (UTC+1), Andrew Godwin
Post by Andrew Godwin
Hi everyone,
For those who haven't seen, Mozilla has awarded $150,000 to Django for
work on Channels and request/response improvements lifted from REST
https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
I'll be coordinating this effort for the most part, but we're still
working out who to fund and the roadmap for the project, as well as work
out how we can pay people for their work on a different scale to the Django
Fellowship, so it might take a bit of time!
I'll be back on here with some questions for people to discuss/answer
about the channels design at some point soon, but a lot of the basic design
is already up at http://channels.readthedocs.org, so take a read over
that if you're interested.
What I can say is that my intention is to both bake Channels into the next
major release of Django, as well as hopefully release a pluggable app
version that will run on 1.8 and 1.9 - I have some plans around how to do
that effectively, but they involve hitherto unforged paths around Django
and how we package dependencies, so I can't say we'll get there 100%.
Andrew
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c25c0c0e-b3be-4190-8798-01ad9238bf25%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Mitar
2015-12-12 13:07:11 UTC
Permalink
Hi!

This is really awesome news! Can I suggest to use WebRTC data channel
instead of Websockets?

I think we should make a paradigm shift and move to use WebRTC instead
of Websockets. This allows many interesting properties:
* connections can be based on SCTP
* have multi-path routing
* you can have data packets which are reliable or unreliable
* you can even have backpressure support (imagine that you could push
back on Meteor publish because client cannot handle observes as
quickly as server is pushing)

There is a great open source project already available which provides
both client server suppor: https://github.com/feross/simple-peer

Some interesting links to read:
* https://github.com/webrtcftw/goals/issues/1
* https://github.com/jbenet/random-ideas/issues/13

I opened a ticket as well: https://github.com/andrewgodwin/channels/issues/38


Mitar
Post by Andrew Godwin
Hi everyone,
For those who haven't seen, Mozilla has awarded $150,000 to Django for work
on Channels and request/response improvements lifted from REST Framework.
https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
I'll be coordinating this effort for the most part, but we're still working
out who to fund and the roadmap for the project, as well as work out how we
can pay people for their work on a different scale to the Django Fellowship,
so it might take a bit of time!
I'll be back on here with some questions for people to discuss/answer about
the channels design at some point soon, but a lot of the basic design is
already up at http://channels.readthedocs.org, so take a read over that if
you're interested.
What I can say is that my intention is to both bake Channels into the next
major release of Django, as well as hopefully release a pluggable app
version that will run on 1.8 and 1.9 - I have some plans around how to do
that effectively, but they involve hitherto unforged paths around Django and
how we package dependencies, so I can't say we'll get there 100%.
Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAFwN1upjXf2LLk2gGA9mzdPQn4DbWehSHNFWhF-GkEk4q54gHA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
http://mitar.tnode.com/
https://twitter.com/mitar_m
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAKLmikNtJhHQntNaSWvyeiz4qhVunM4OZ%2BJM-9v4VpuArmfsdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Aymeric Augustin
2015-12-12 14:29:29 UTC
Permalink
Hello,

Unless I missed something, channels aren’t about websockets. They add asynchronous processing capabilities to Django. They aim at being compatible with any bidirectional protocol.

Django should support common protocols out of the box and document a public API for third-party projects to hook to and support additional protocols.

At this point, I’m not convinced that simple-peer is sufficiently mainstream to be the recommended solution for Django websites.

Best regards,
--
Aymeric.
Post by Mitar
Hi!
This is really awesome news! Can I suggest to use WebRTC data channel
instead of Websockets?
I think we should make a paradigm shift and move to use WebRTC instead
* connections can be based on SCTP
* have multi-path routing
* you can have data packets which are reliable or unreliable
* you can even have backpressure support (imagine that you could push
back on Meteor publish because client cannot handle observes as
quickly as server is pushing)
There is a great open source project already available which provides
both client server suppor: https://github.com/feross/simple-peer
* https://github.com/webrtcftw/goals/issues/1
* https://github.com/jbenet/random-ideas/issues/13
I opened a ticket as well: https://github.com/andrewgodwin/channels/issues/38
Mitar
Post by Andrew Godwin
Hi everyone,
For those who haven't seen, Mozilla has awarded $150,000 to Django for work
on Channels and request/response improvements lifted from REST Framework.
https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
I'll be coordinating this effort for the most part, but we're still working
out who to fund and the roadmap for the project, as well as work out how we
can pay people for their work on a different scale to the Django Fellowship,
so it might take a bit of time!
I'll be back on here with some questions for people to discuss/answer about
the channels design at some point soon, but a lot of the basic design is
already up at http://channels.readthedocs.org, so take a read over that if
you're interested.
What I can say is that my intention is to both bake Channels into the next
major release of Django, as well as hopefully release a pluggable app
version that will run on 1.8 and 1.9 - I have some plans around how to do
that effectively, but they involve hitherto unforged paths around Django and
how we package dependencies, so I can't say we'll get there 100%.
Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAFwN1upjXf2LLk2gGA9mzdPQn4DbWehSHNFWhF-GkEk4q54gHA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
http://mitar.tnode.com/
https://twitter.com/mitar_m
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAKLmikNtJhHQntNaSWvyeiz4qhVunM4OZ%2BJM-9v4VpuArmfsdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/490B7FFD-8062-4385-82AB-1138A0D6DDE4%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Mitar
2015-12-12 22:10:07 UTC
Permalink
Hi!

On Sat, Dec 12, 2015 at 6:29 AM, Aymeric Augustin
Post by Aymeric Augustin
Unless I missed something, channels aren’t about websockets. They add asynchronous processing capabilities to Django. They aim at being compatible with any bidirectional protocol.
Yes, but currently, that "bidirectional protocol" is "bidirectional
protocol without backpressure" which makes it really useless in any
serious production use with scaling. It is interesting for smaller toy
use cases, but without backpressure you simply cannot handle
congestion and your systems blow in your user's face. node.js has
learned this the hard way by reimplementing their streaming API
multiple times.

What I would propose is that Django officially exposes channels API
with backpressure and then for simpler projects (or one where you do
not care about delivery) one can use Websockets, but otherwise for
production use for heavy loads WebRTC data channels with backpressure
should be provided.
Post by Aymeric Augustin
At this point, I’m not convinced that simple-peer is sufficiently mainstream to be the recommended solution for Django websites.
No, that is just an example of the node.js module which provides
server-side WebRTC support.

And yes, this is something which is cutting edge. I would be
completely OK by supporting Websockets now on the transport level, but
the API level should understand and support backpressure. Otherwise we
will have issues in the future once this feature gets widely deployed
and started using in production.


Mitar
--
http://mitar.tnode.com/
https://twitter.com/mitar_m
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAKLmikPjYgB3LUsSrDvxQ20rtopAU21meC6xNLRRjcx1y%3DY76Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Aymeric Augustin
2015-12-13 08:55:11 UTC
Permalink
Post by Mitar
I would be
completely OK by supporting Websockets now on the transport level, but
the API level should understand and support backpressure.
Hello Mitar,

Yes, I fully agree with this way to frame the topic.

I'm don't know how this would manifest in the core API nor how much thought Andrew has given to this issue, but I'm interested in making sure we get it right. Thanks for bringing it up!
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/06F12986-4716-4745-BF0D-5560E45043BA%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
b***@rentlytics.com
2015-12-15 21:35:25 UTC
Permalink
Hey,

Channels sounds a lot like celery (or celery sounds like part of
channels). Is that a fair read? Looking for tighter REST integration
either way.

Thanks,
-Ben
Post by Andrew Godwin
Hi everyone,
For those who haven't seen, Mozilla has awarded $150,000 to Django for
work on Channels and request/response improvements lifted from REST
https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
I'll be coordinating this effort for the most part, but we're still
working out who to fund and the roadmap for the project, as well as work
out how we can pay people for their work on a different scale to the Django
Fellowship, so it might take a bit of time!
I'll be back on here with some questions for people to discuss/answer
about the channels design at some point soon, but a lot of the basic design
is already up at http://channels.readthedocs.org, so take a read over
that if you're interested.
What I can say is that my intention is to both bake Channels into the next
major release of Django, as well as hopefully release a pluggable app
version that will run on 1.8 and 1.9 - I have some plans around how to do
that effectively, but they involve hitherto unforged paths around Django
and how we package dependencies, so I can't say we'll get there 100%.
Andrew
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/299ede52-6ae4-46d3-8667-40e7fa6a89a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Aymeric Augustin
2015-12-15 22:09:06 UTC
Permalink
Hi Ben,

Celery and channels don’t tackle the same problem.

Celery is an asynchronous task queue. It is designed perform expensive work after responding to a HTTP request. At that point there is no possible communication with the browser.

Channels is an asynchronous message framework. It enables asynchronous messaging between the browser and Django, going beyond the current request-response structure.

Best regards,
--
Aymeric.
Hey,
Channels sounds a lot like celery (or celery sounds like part of channels). Is that a fair read? Looking for tighter REST integration either way.
Thanks,
-Ben
Hi everyone,
For those who haven't seen, Mozilla has awarded $150,000 to Django for work on Channels and request/response improvements lifted from REST Framework. More in the blog post here: https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/ <https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/>
I'll be coordinating this effort for the most part, but we're still working out who to fund and the roadmap for the project, as well as work out how we can pay people for their work on a different scale to the Django Fellowship, so it might take a bit of time!
I'll be back on here with some questions for people to discuss/answer about the channels design at some point soon, but a lot of the basic design is already up at http://channels.readthedocs.org <http://channels.readthedocs.org/>, so take a read over that if you're interested.
What I can say is that my intention is to both bake Channels into the next major release of Django, as well as hopefully release a pluggable app version that will run on 1.8 and 1.9 - I have some plans around how to do that effectively, but they involve hitherto unforged paths around Django and how we package dependencies, so I can't say we'll get there 100%.
Andrew
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
Visit this group at https://groups.google.com/group/django-developers <https://groups.google.com/group/django-developers>.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/299ede52-6ae4-46d3-8667-40e7fa6a89a6%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/299ede52-6ae4-46d3-8667-40e7fa6a89a6%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/D5557CBB-03D8-4DB2-AAFA-B4063E227D8A%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Ben Liyanage
2015-12-15 23:07:16 UTC
Permalink
So a channel must result in a response to a browser? It seems both from
the config (routing things into functions, or queues) and the underlying
tech (using redis or whatever) sounds very similar to celery.

From the docs:

- Interface servers, which communicate between Django and the outside
world. This includes a WSGI adapter as well as a separate WebSocket server
- we’ll cover this later.
- The channel backend, which is a combination of pluggable Python code
and a datastore (a database, or Redis) and responsible for transporting
messages.
- The workers, that listen on all relevant channels and run consumer
code when a message is ready.

I get that the goal of this is for asynchronous web requests, but if it's
generalized right it seems like it could cover doing any kind of work
without the pressure of completing a web transaction in a timely fashion.

-Ben

On Tue, Dec 15, 2015 at 2:09 PM, Aymeric Augustin <
Post by Aymeric Augustin
Hi Ben,
Celery and channels don’t tackle the same problem.
Celery is an asynchronous task queue. It is designed perform expensive
work after responding to a HTTP request. At that point there is no possible
communication with the browser.
Channels is an asynchronous message framework. It enables asynchronous
messaging between the browser and Django, going beyond the current
request-response structure.
Best regards,
--
Aymeric.
Hey,
Channels sounds a lot like celery (or celery sounds like part of
channels). Is that a fair read? Looking for tighter REST integration
either way.
Thanks,
-Ben
Post by Andrew Godwin
Hi everyone,
For those who haven't seen, Mozilla has awarded $150,000 to Django for
work on Channels and request/response improvements lifted from REST
https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/
I'll be coordinating this effort for the most part, but we're still
working out who to fund and the roadmap for the project, as well as work
out how we can pay people for their work on a different scale to the Django
Fellowship, so it might take a bit of time!
I'll be back on here with some questions for people to discuss/answer
about the channels design at some point soon, but a lot of the basic design
is already up at http://channels.readthedocs.org, so take a read over
that if you're interested.
What I can say is that my intention is to both bake Channels into the
next major release of Django, as well as hopefully release a pluggable app
version that will run on 1.8 and 1.9 - I have some plans around how to do
that effectively, but they involve hitherto unforged paths around Django
and how we package dependencies, so I can't say we'll get there 100%.
Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/299ede52-6ae4-46d3-8667-40e7fa6a89a6%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/299ede52-6ae4-46d3-8667-40e7fa6a89a6%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/django-developers/3pTMfHa8SFE/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/D5557CBB-03D8-4DB2-AAFA-B4063E227D8A%40polytechnique.org
<https://groups.google.com/d/msgid/django-developers/D5557CBB-03D8-4DB2-AAFA-B4063E227D8A%40polytechnique.org?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
*Ben Liyanage *|* Software Engineer *|* Rentlytics, Inc.*
Phone: (410) 336-2464 | Email: ***@rentlytics.com
1132 Howard Street, San Francisco CA 94107
Visit our Website <http://www.rentlytics.com> | Watch our Video

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CADgLBUOdzxx1LxprsggcYKH22SctV%3DQsqLVsr-yJCspQ11gEbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Shai Berger
2015-12-15 23:41:09 UTC
Permalink
Post by Ben Liyanage
I get that the goal of this is for asynchronous web requests, but if it's
generalized right it seems like it could cover doing any kind of work
without the pressure of completing a web transaction in a timely fashion.
No, because Channels are designed to be non-reliable (each message is
delivered at most once), and if I understand correctly this is a key point in
the design. Celery gives you much stronger delivery guarantees,

Shai.
Ben Liyanage
2015-12-16 01:05:06 UTC
Permalink
And, beyond that, there are plenty of non-critical tasks that applications
could easily offload until after a response has been sent - like saving
things into a cache or thumbnailing newly-uploaded images.
http://channels.readthedocs.org/en/latest/concepts.html#concepts

I mean this example sounds like celery to me. You don't want an image to
maybe have a thumbnail generated. And you have no return value to the
client when the thumbnail is generated.

-Ben
Post by Ben Liyanage
I get that the goal of this is for asynchronous web requests, but if it's
generalized right it seems like it could cover doing any kind of work
without the pressure of completing a web transaction in a timely fashion.
No, because Channels are designed to be non-reliable (each message is
delivered at most once), and if I understand correctly this is a key point in
the design. Celery gives you much stronger delivery guarantees,
Shai.
--
*Ben Liyanage *|* Software Engineer *|* Rentlytics, Inc.*
Phone: (410) 336-2464 | Email: ***@rentlytics.com
1132 Howard Street, San Francisco CA 94107
Visit our Website <http://www.rentlytics.com> | Watch our Video
http://youtu.be/Pe_9KE_fj34
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CADgLBUOb8v0VozfcX%3D7t8_zgGznbqHU%3D57PZpoArP5%3DGC8yxQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Curtis Maloney
2015-12-16 04:15:49 UTC
Permalink
And, beyond that, there are plenty of non-critical tasks that
applications could easily offload until after a response has been
sent - like saving things into a cache or thumbnailing
newly-uploaded images.
http://channels.readthedocs.org/en/latest/concepts.html#concepts
I mean this example sounds like celery to me. You don't want an image
to maybe have a thumbnail generated. And you have no return value to
the client when the thumbnail is generated.
There are certainly some cases, AIUI, where channels can obviate the
need for Celery... like sending emails, or thumbnailing uploaded image,
and so on.

However, given the different reliability profiles, there's still room
for both.

--
Curtis
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5670E575.9020106%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.
Markus Holtermann
2015-12-16 06:08:59 UTC
Permalink
From my point of view the major non-functional difference is the interaction with a client in *some way*. While a celery task runs without any form of client, a channels method would mostly do that. Picture the following example:

A user uploads an image. This could happen through the common request-response cycle. When the upload is done the server creates a celery task to generate a bunch of thumbnails and returns a http response. At this point the image is already available online but the thumbnails aren't. Every client could start a websocket connection through channels to get notified once a thumbnail is available. This notification is send through a server side callback called by the celery task.

/Markus
Post by Curtis Maloney
And, beyond that, there are plenty of non-critical tasks that
applications could easily offload until after a response has been
sent - like saving things into a cache or thumbnailing
newly-uploaded images.
http://channels.readthedocs.org/en/latest/concepts.html#concepts
I mean this example sounds like celery to me. You don't want an
image
to maybe have a thumbnail generated. And you have no return value to
the client when the thumbnail is generated.
There are certainly some cases, AIUI, where channels can obviate the
need for Celery... like sending emails, or thumbnailing uploaded image,
and so on.
However, given the different reliability profiles, there's still room
for both.
--
Curtis
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6037EBAB-C6F1-4A22-A632-A89812B53A4B%40markusholtermann.eu.
For more options, visit https://groups.google.com/d/optout.
Shai Berger
2015-12-16 07:31:58 UTC
Permalink
Post by Markus Holtermann
From my point of view the major non-functional difference is the
interaction with a client in *some way*. While a celery task runs without
any form of client, a channels method would mostly do that. Picture the
A user uploads an image. This could happen through the common
request-response cycle. When the upload is done the server creates a
celery task to generate a bunch of thumbnails and returns a http response.
But here is Ben's point: Why would you introduce an additional moving part
here (Celery), when the Channels documentation specifically mentions this as a
use-case?

If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.

Shai.
Andrew Godwin
2015-12-16 09:35:31 UTC
Permalink
Channels is indeed a replacement for Celery if you're OK with not having
delays, retry, or the other advanced options Celery has - Channels just
requires an event to respond to, and this can be triggered by the user
during a view much like a Celery task.

But yes, channels deliberately has an at-most-once delivery style, which
will work for operations that aren't critical, or if people build their own
retry logic (but at that point, maybe use Celery).

Andrew
Post by Markus Holtermann
Post by Markus Holtermann
From my point of view the major non-functional difference is the
interaction with a client in *some way*. While a celery task runs without
any form of client, a channels method would mostly do that. Picture the
A user uploads an image. This could happen through the common
request-response cycle. When the upload is done the server creates a
celery task to generate a bunch of thumbnails and returns a http
response.
But here is Ben's point: Why would you introduce an additional moving part
here (Celery), when the Channels documentation specifically mentions this as a
use-case?
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
Shai.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uoyJT8r9kL4-rYNqyzQD2LC_9Y%3DaDjbPisA3GYoKzua3A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Markus Holtermann
2015-12-16 09:52:54 UTC
Permalink
Post by Mitar
Post by Markus Holtermann
From my point of view the major non-functional difference is the
interaction with a client in *some way*. While a celery task runs
without
Post by Markus Holtermann
any form of client, a channels method would mostly do that. Picture
the
Post by Markus Holtermann
A user uploads an image. This could happen through the common
request-response cycle. When the upload is done the server creates a
celery task to generate a bunch of thumbnails and returns a http
response.
But here is Ben's point: Why would you introduce an additional moving part
here (Celery), when the Channels documentation specifically mentions this as a
use-case?
Because my "generate a thumbnail" might me some video decoding or another long running process that would block possible further requests, or simply because I could easier reuse that particular task in other parts of the website as well.
Post by Mitar
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".

/Markus
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/D484E906-05BC-4959-A786-9F80BFF41B5C%40markusholtermann.eu.
For more options, visit https://groups.google.com/d/optout.
Andrew Godwin
2015-12-16 11:22:25 UTC
Permalink
Post by Markus Holtermann
Post by Shai Berger
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
I don't agree; I think it's more "do you have enough workers to handle
this". Channels makes all workers serve all requests, so if you're doing
e.g. video encoding you can't segregate it from request handling. However,
you could have multiple channel backends and thus multiple worker pools to
achieve this.

Andrew
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uo58xzT_yt5PoZ47RMu-S3biXjVLggned-_TS4R_ba80g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Javier Guerra Giraldez
2015-12-16 17:17:44 UTC
Permalink
On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann
Post by Markus Holtermann
Post by Shai Berger
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
as i read it:

celery has an specific use pattern: the Django process emits a
message to some other process. It doesn't alter the request/response
cycle in any way. the most common use is to initiate a 'background
process' from the request-bound Django process to another, non-Django
process.


The "channels" design makes Django a processor of messages, the first
example is a 'normal' http request, which would be just a message from
the user. another example is asynchronous request/responses,
typically WebSockets, SSE, etc. finally, it could also allow a
Django process on the receiving end of an intra-server message. A
future version of Celery could use that as a new infrastructure
option.
--
Javier
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFkDaoRiRJiOUQK-xCi8kwkrR%3DV9K6rMw%2BstZ16ZOrNn7%3D_0Mg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Samuel Bishop
2015-12-17 10:30:16 UTC
Permalink
I'm uncertain how popular the suggestion will be but ... are "channels" the
right solution to our async/concurrency problems? ( I'm all for
django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the explanations
I'm reading as I follow along with the progress of 'Django Channels", just
feels "less good" than solving this problem via Actor Model Concurrency
(championed by Erlang, Elixir, Stackless Python, etc).

There is an excellent actor model concurrency library for python that
already supports 'async django'. "Pulsar"
(http://pythonhosted.org/pulsar/index.html) and the documentation here
http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways to
use actor model concurrency to attack the issue of using Django with
asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels
sounds very similar to what they provide to work around the limitations of
django's blocking request processing cycle other parts sound much more
complicated, for instance the need for various back ends
in-memory/redis/etc.

I know Pulsar is currently Python 3.4 and newer only, but in the past it
supported 2.7, so anything we can learn from their concurrency approach can
work on Python 2.7 to maintain our backwards compatibility going forward.
You can see an example of Pulsar using python 2 if you look on the
benchmark page for their python powered 'redis clone'
- https://gist.github.com/lsbardel/8068579 - One of the benchmarks is using
pypy so the code definitely worked in a python 2.7 environment.

Would the work to integrate channels as part of Django improve the ability
to run Django in a completely async mode and enhance its performance when
run under Pulsar? or would it just be something that needed to be turned
off if we wanted to use something like Pulsar to power our "async django".
Is there anything to be gained from using a more actor driven concurrency
model or adopting any of the other methods used in their 'pulse' django app?
I suppose I'm just trying to work out how to compare using Django-Channels
with using Pulse from the Pulsar project.
On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
Post by Markus Holtermann
Post by Shai Berger
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
I don't agree; I think it's more "do you have enough workers to handle
this". Channels makes all workers serve all requests, so if you're doing
e.g. video encoding you can't segregate it from request handling. However,
you could have multiple channel backends and thus multiple worker pools to
achieve this.
Andrew
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Andrew Godwin
2015-12-17 11:07:37 UTC
Permalink
Hi Samuel,

You'll notice if you look carefully that I avoid saying "async" almost
anywhere in the Channels announcements or documentation, and when I do,
it's qualified very carefully. Channels makes Django event-driven, but it
does not provide full "in-process" async like Twisted, asyncio, or
something like Pulsar.

Channels is actually kind of related to the actor model in some ways, but
it's simpler; imagine it as a pool of identical actors that are all
stateless, apart from the interface servers, which have the incoming socket
and thus the state.

The key thing is that Channels does nothing to make Django asynchronous.
All the Django code will remain synchronous (the ORM, templating, etc.),
and run in synchronous processes that just process one message after the
next. The only part of the system that will require an async framework is
the interface servers, as to be anywhere near efficient they need to serve
more than one socket at once; right now we're using Twisted and asyncio for
those, purely as they're incredibly mature and well-supported by the
community, but you could swap out anything there that achieved the same
goal (greenlets, for example, though I personally dislike their
implicitness).

Channels essentially takes the worker-prefork model that's been used in
webservers for years and applies it to a lower-level message concept rather
than HTTP requests (http requests are just one kind of message). This not
only has the advantage of a lot of previous work and research into
architecting and scaling these kinds of systems, it also means the actual
amount of work in core Django isn't crazy; it's more about providing nice
places to wrap up the worker loop and nice utilities around authentication
and session management.

If you read through the docs there's a bit more about this, but basically,
the fundamental design goal here is that Channels be no harder to use than
normal Django; indeed, it's almost exactly the same view concept, but
instead of a request you get a message, and the newer developers I've
interviewed about it seem to be comfortable with the extension of the
current abstraction straight away. I don't ever want people to have to
write asynchronous code if they don't want to; it's currently so easy to
shoot yourself in the foot in almost any Python async solution that I don't
think that makes for a good developer experience.

The "complexity" of the message backends you describe is because they're
network-transparent; there is indeed a simple, in-process one that uses
threads too, but the whole point of this design is that you run Django over
multiple processes on many machines, and in order to get something like
that you need a place to funnel messages through (another potential backend
design could use direct connection and process discovery, perhaps, or an
Erlang-like process spawn-and-track design with an overarching controller
process with access to many pre-selected machines).

The idea of making an asynchronous Django is tangential to making Channels;
this is more about changing the design abstraction of Django so we can fit
the concept in, and then providing a neat solution that drops into the gap.
There's nothing stopping someone making the worker processes for Channels
async-aware as well, so they can run multiple message consumers at once,
but that's such a complex and risk-fraught task that I don't think it's
worth it when we can get 90% of the benefits with a much simpler and
easier-to-design-and-understand system. If you want to see an example of
actual async Django, see Amber's talk from Django Under the Hood this year,
where she managed to get it working decently well under Twisted.

I expect Channels to smooth out Django's performance mostly as a
side-effect of the worker model providing demand-based load balancing by
design, but I doubt it will be a huge increase until people start using the
primitives it provides to offload tasks to other workers.

I hope that somewhat long-winded explanation helps - if not, let me know
what's still unclear and I'll try to tackle it. Clearing up what we're
trying to do and communicating it is important to me; I want to make sure
everyone knows so we can get good feedback on this stuff. For what it's
worth, I did consider the actor model along with several other kinds of
concurrency approaches (I've been prototyping bits of this for years); this
approach ended up being the nicest design and the one I have the most
confidence in that we can scale and shard to large work volumes.

Andrew
Post by Samuel Bishop
I'm uncertain how popular the suggestion will be but ... are "channels"
the right solution to our async/concurrency problems? ( I'm all for
django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the
explanations I'm reading as I follow along with the progress of 'Django
Channels", just feels "less good" than solving this problem via Actor Model
Concurrency (championed by Erlang, Elixir, Stackless Python, etc).
There is an excellent actor model concurrency library for python that
already supports 'async django'. "Pulsar" (
http://pythonhosted.org/pulsar/index.html) and the documentation here
http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways
to use actor model concurrency to attack the issue of using Django with
asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels
sounds very similar to what they provide to work around the limitations of
django's blocking request processing cycle other parts sound much more
complicated, for instance the need for various back ends
in-memory/redis/etc.
I know Pulsar is currently Python 3.4 and newer only, but in the past it
supported 2.7, so anything we can learn from their concurrency approach can
work on Python 2.7 to maintain our backwards compatibility going forward.
You can see an example of Pulsar using python 2 if you look on the
benchmark page for their python powered 'redis clone' -
https://gist.github.com/lsbardel/8068579 - One of the benchmarks is using
pypy so the code definitely worked in a python 2.7 environment.
Would the work to integrate channels as part of Django improve the ability
to run Django in a completely async mode and enhance its performance when
run under Pulsar? or would it just be something that needed to be turned
off if we wanted to use something like Pulsar to power our "async django".
Is there anything to be gained from using a more actor driven concurrency
model or adopting any of the other methods used in their 'pulse' django app?
I suppose I'm just trying to work out how to compare using Django-Channels
with using Pulse from the Pulsar project.
On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
Post by Markus Holtermann
Post by Shai Berger
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
I don't agree; I think it's more "do you have enough workers to handle
this". Channels makes all workers serve all requests, so if you're doing
e.g. video encoding you can't segregate it from request handling. However,
you could have multiple channel backends and thus multiple worker pools to
achieve this.
Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upsqkL9_qN6NfO7PB9xjHMaPrjL9whwpnmPo3Ru4ZbO0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Samuel Bishop
2015-12-21 06:05:39 UTC
Permalink
Hi Andrew,

Thanks for the detailed response. I'm glad to hear you have tried a few
different ways to implement this, and having read through your reply, and
the channels docs again, I have more questions :-)

Your point about asynchronous code introducing needless complexity is very
well founded, its one of the reasons I like Actor Model Concurrency over
other concurrency models, it scales well computationally, and *mentally*,
as a programmer, I am trying break down my real world goals into discrete
steps of computations, comparisons, iterations and by combining these
steps, I construct my program. Actor Model Concurrency, is no more
complicated than saying "This step can be done without affecting anything
else thats going on", which makes the 'actor/task' much easier to introduce
to existing synchronous code than most other concurrency methods.

One thing that sticks with me as I think about the current channels design,
relates to the idea I've raised in another discussion (
https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ).
If I put my architect hat on and think about the future a bit, a 'more'
actor based design to channels might simplify later additional async work.
Since all the websocket stuff is new, I'll stick to the existing HTTP cycle
in this example. In version 1 a 'vanilla' Django task/actor would be a full
HTTP request response cycle, like how Django and WSGI currently work. Our
Django task is effectively our WSGI task, much like Pulsar Pulse. If all of
Django is contained inside our new task/worker/supervisor Actor layer, then
it becomes fairly trivial to do things like create separate request
handler, database query, and template rendering task/actors and chain their
calls. Since in the previously linked to discussion thread, I'm talking
about decoupling the ORM from the 'upper' layers of Django, its hard to not
also think what opportunities that would open up for async efforts. How do
you see the architecture of Django channels accommodating such improvements
in future? At the moment your focused on keeping things compatible and
avoiding the need for any of the existing synchronous Django code to
change, is this introducing any design tradeoffs that you are aware of with
regards to hypothetical future developments?

Django Channels feels like it contains a possible asynchronous replacement
for signals but is 'holding back', many cases where I can see myself using
a channel, I'm using them with a signal handler, can you elaborate a bit on
your thoughts here? I assume its a compatibility decision, but is there
more to it than just that?

Regards
Sam
Post by Andrew Godwin
Hi Samuel,
You'll notice if you look carefully that I avoid saying "async" almost
anywhere in the Channels announcements or documentation, and when I do,
it's qualified very carefully. Channels makes Django event-driven, but it
does not provide full "in-process" async like Twisted, asyncio, or
something like Pulsar.
Channels is actually kind of related to the actor model in some ways, but
it's simpler; imagine it as a pool of identical actors that are all
stateless, apart from the interface servers, which have the incoming socket
and thus the state.
The key thing is that Channels does nothing to make Django asynchronous.
All the Django code will remain synchronous (the ORM, templating, etc.),
and run in synchronous processes that just process one message after the
next. The only part of the system that will require an async framework is
the interface servers, as to be anywhere near efficient they need to serve
more than one socket at once; right now we're using Twisted and asyncio for
those, purely as they're incredibly mature and well-supported by the
community, but you could swap out anything there that achieved the same
goal (greenlets, for example, though I personally dislike their
implicitness).
Channels essentially takes the worker-prefork model that's been used in
webservers for years and applies it to a lower-level message concept rather
than HTTP requests (http requests are just one kind of message). This not
only has the advantage of a lot of previous work and research into
architecting and scaling these kinds of systems, it also means the actual
amount of work in core Django isn't crazy; it's more about providing nice
places to wrap up the worker loop and nice utilities around authentication
and session management.
If you read through the docs there's a bit more about this, but basically,
the fundamental design goal here is that Channels be no harder to use than
normal Django; indeed, it's almost exactly the same view concept, but
instead of a request you get a message, and the newer developers I've
interviewed about it seem to be comfortable with the extension of the
current abstraction straight away. I don't ever want people to have to
write asynchronous code if they don't want to; it's currently so easy to
shoot yourself in the foot in almost any Python async solution that I don't
think that makes for a good developer experience.
The "complexity" of the message backends you describe is because they're
network-transparent; there is indeed a simple, in-process one that uses
threads too, but the whole point of this design is that you run Django over
multiple processes on many machines, and in order to get something like
that you need a place to funnel messages through (another potential backend
design could use direct connection and process discovery, perhaps, or an
Erlang-like process spawn-and-track design with an overarching controller
process with access to many pre-selected machines).
The idea of making an asynchronous Django is tangential to making
Channels; this is more about changing the design abstraction of Django so
we can fit the concept in, and then providing a neat solution that drops
into the gap. There's nothing stopping someone making the worker processes
for Channels async-aware as well, so they can run multiple message
consumers at once, but that's such a complex and risk-fraught task that I
don't think it's worth it when we can get 90% of the benefits with a much
simpler and easier-to-design-and-understand system. If you want to see an
example of actual async Django, see Amber's talk from Django Under the Hood
this year, where she managed to get it working decently well under Twisted.
I expect Channels to smooth out Django's performance mostly as a
side-effect of the worker model providing demand-based load balancing by
design, but I doubt it will be a huge increase until people start using the
primitives it provides to offload tasks to other workers.
I hope that somewhat long-winded explanation helps - if not, let me know
what's still unclear and I'll try to tackle it. Clearing up what we're
trying to do and communicating it is important to me; I want to make sure
everyone knows so we can get good feedback on this stuff. For what it's
worth, I did consider the actor model along with several other kinds of
concurrency approaches (I've been prototyping bits of this for years); this
approach ended up being the nicest design and the one I have the most
confidence in that we can scale and shard to large work volumes.
Andrew
Post by Samuel Bishop
I'm uncertain how popular the suggestion will be but ... are "channels"
the right solution to our async/concurrency problems? ( I'm all for
django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the
explanations I'm reading as I follow along with the progress of 'Django
Channels", just feels "less good" than solving this problem via Actor Model
Concurrency (championed by Erlang, Elixir, Stackless Python, etc).
There is an excellent actor model concurrency library for python that
already supports 'async django'. "Pulsar" (
http://pythonhosted.org/pulsar/index.html) and the documentation here
http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways
to use actor model concurrency to attack the issue of using Django with
asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels
sounds very similar to what they provide to work around the limitations of
django's blocking request processing cycle other parts sound much more
complicated, for instance the need for various back ends
in-memory/redis/etc.
I know Pulsar is currently Python 3.4 and newer only, but in the past it
supported 2.7, so anything we can learn from their concurrency approach can
work on Python 2.7 to maintain our backwards compatibility going forward.
You can see an example of Pulsar using python 2 if you look on the
benchmark page for their python powered 'redis clone' -
https://gist.github.com/lsbardel/8068579 - One of the benchmarks is
using pypy so the code definitely worked in a python 2.7 environment.
Would the work to integrate channels as part of Django improve the
ability to run Django in a completely async mode and enhance its
performance when run under Pulsar? or would it just be something that
needed to be turned off if we wanted to use something like Pulsar to power
our "async django". Is there anything to be gained from using a more actor
driven concurrency model or adopting any of the other methods used in their
'pulse' django app?
I suppose I'm just trying to work out how to compare using
Django-Channels with using Pulse from the Pulsar project.
On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
Post by Markus Holtermann
Post by Shai Berger
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
I don't agree; I think it's more "do you have enough workers to handle
this". Channels makes all workers serve all requests, so if you're doing
e.g. video encoding you can't segregate it from request handling. However,
you could have multiple channel backends and thus multiple worker pools to
achieve this.
Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Andrew Godwin
2015-12-21 11:31:17 UTC
Permalink
Post by Samuel Bishop
Hi Andrew,
Thanks for the detailed response. I'm glad to hear you have tried a few
different ways to implement this, and having read through your reply, and
the channels docs again, I have more questions :-)
Your point about asynchronous code introducing needless complexity is very
well founded, its one of the reasons I like Actor Model Concurrency over
other concurrency models, it scales well computationally, and *mentally*,
as a programmer, I am trying break down my real world goals into discrete
steps of computations, comparisons, iterations and by combining these
steps, I construct my program. Actor Model Concurrency, is no more
complicated than saying "This step can be done without affecting anything
else thats going on", which makes the 'actor/task' much easier to introduce
to existing synchronous code than most other concurrency methods.
I agree - I'm a big fan of actors in general and it's usually what I turn
to to write complex simulations when I'm mucking around with game backends.
Post by Samuel Bishop
One thing that sticks with me as I think about the current channels
design, relates to the idea I've raised in another discussion (
https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ).
If I put my architect hat on and think about the future a bit, a 'more'
actor based design to channels might simplify later additional async work.
Since all the websocket stuff is new, I'll stick to the existing HTTP cycle
in this example. In version 1 a 'vanilla' Django task/actor would be a full
HTTP request response cycle, like how Django and WSGI currently work. Our
Django task is effectively our WSGI task, much like Pulsar Pulse. If all of
Django is contained inside our new task/worker/supervisor Actor layer, then
it becomes fairly trivial to do things like create separate request
handler, database query, and template rendering task/actors and chain their
calls. Since in the previously linked to discussion thread, I'm talking
about decoupling the ORM from the 'upper' layers of Django, its hard to not
also think what opportunities that would open up for async efforts. How do
you see the architecture of Django channels accommodating such improvements
in future? At the moment your focused on keeping things compatible and
avoiding the need for any of the existing synchronous Django code to
change, is this introducing any design tradeoffs that you are aware of with
regards to hypothetical future developments?
So, what you propose obviously isn't feasible in the short term - I'm doing
as much as I can to decouple Channels from "rewrite all of Django" - but
notably, channels is a subset of the actor model. In particular, you could
see consumers as stateless, single-action actors - but, this means that any
future abstraction would be able to build on that.
Post by Samuel Bishop
Django Channels feels like it contains a possible asynchronous replacement
for signals but is 'holding back', many cases where I can see myself using
a channel, I'm using them with a signal handler, can you elaborate a bit on
your thoughts here? I assume its a compatibility decision, but is there
more to it than just that?
It's entirely a compatibility decision - Channels needs to ship in a way
where code written for 1.9 will work the same, barring a few edge cases
where people dip down to raw WSGI.

You can entirely replace signals with custom channels, and I want to ship
example code that does so (it's two lines of code, pretty much), but they
work in a different way - for example, right now the pre_save signal can
actually prevent a save from occurring, whereas if it were on a channel
we'd have to then link the save consumer up to be run as a result of
pre_save, which starts down a path of compatibility nightmares.

Channels should be flexible enough that if you want to pursue the actor
pattern you can - the underlying library basically just has send() and
receive_many([channel_name, ...]) methods, so you can see how that could be
used to make different task-oriented processes rather than the generic
worker model that's going in now - it's just a case of keeping a major
addition like this as simple and backwards-compatible as possible.
Rewriting everything from scratch would be lovely, but unfortunately part
of maintaining a framework is not doing that.

Part of me, though, would be very interested in such a project being
undertaken to see how far it could go, though I'm not sure you could then
call the end result Django - at least, not without getting there
incrementally over many releases.

Andrew
Post by Samuel Bishop
Post by Andrew Godwin
Hi Samuel,
You'll notice if you look carefully that I avoid saying "async" almost
anywhere in the Channels announcements or documentation, and when I do,
it's qualified very carefully. Channels makes Django event-driven, but it
does not provide full "in-process" async like Twisted, asyncio, or
something like Pulsar.
Channels is actually kind of related to the actor model in some ways, but
it's simpler; imagine it as a pool of identical actors that are all
stateless, apart from the interface servers, which have the incoming socket
and thus the state.
The key thing is that Channels does nothing to make Django asynchronous.
All the Django code will remain synchronous (the ORM, templating, etc.),
and run in synchronous processes that just process one message after the
next. The only part of the system that will require an async framework is
the interface servers, as to be anywhere near efficient they need to serve
more than one socket at once; right now we're using Twisted and asyncio for
those, purely as they're incredibly mature and well-supported by the
community, but you could swap out anything there that achieved the same
goal (greenlets, for example, though I personally dislike their
implicitness).
Channels essentially takes the worker-prefork model that's been used in
webservers for years and applies it to a lower-level message concept rather
than HTTP requests (http requests are just one kind of message). This not
only has the advantage of a lot of previous work and research into
architecting and scaling these kinds of systems, it also means the actual
amount of work in core Django isn't crazy; it's more about providing nice
places to wrap up the worker loop and nice utilities around authentication
and session management.
If you read through the docs there's a bit more about this, but
basically, the fundamental design goal here is that Channels be no harder
to use than normal Django; indeed, it's almost exactly the same view
concept, but instead of a request you get a message, and the newer
developers I've interviewed about it seem to be comfortable with the
extension of the current abstraction straight away. I don't ever want
people to have to write asynchronous code if they don't want to; it's
currently so easy to shoot yourself in the foot in almost any Python async
solution that I don't think that makes for a good developer experience.
The "complexity" of the message backends you describe is because they're
network-transparent; there is indeed a simple, in-process one that uses
threads too, but the whole point of this design is that you run Django over
multiple processes on many machines, and in order to get something like
that you need a place to funnel messages through (another potential backend
design could use direct connection and process discovery, perhaps, or an
Erlang-like process spawn-and-track design with an overarching controller
process with access to many pre-selected machines).
The idea of making an asynchronous Django is tangential to making
Channels; this is more about changing the design abstraction of Django so
we can fit the concept in, and then providing a neat solution that drops
into the gap. There's nothing stopping someone making the worker processes
for Channels async-aware as well, so they can run multiple message
consumers at once, but that's such a complex and risk-fraught task that I
don't think it's worth it when we can get 90% of the benefits with a much
simpler and easier-to-design-and-understand system. If you want to see an
example of actual async Django, see Amber's talk from Django Under the Hood
this year, where she managed to get it working decently well under Twisted.
I expect Channels to smooth out Django's performance mostly as a
side-effect of the worker model providing demand-based load balancing by
design, but I doubt it will be a huge increase until people start using the
primitives it provides to offload tasks to other workers.
I hope that somewhat long-winded explanation helps - if not, let me know
what's still unclear and I'll try to tackle it. Clearing up what we're
trying to do and communicating it is important to me; I want to make sure
everyone knows so we can get good feedback on this stuff. For what it's
worth, I did consider the actor model along with several other kinds of
concurrency approaches (I've been prototyping bits of this for years); this
approach ended up being the nicest design and the one I have the most
confidence in that we can scale and shard to large work volumes.
Andrew
Post by Samuel Bishop
I'm uncertain how popular the suggestion will be but ... are "channels"
the right solution to our async/concurrency problems? ( I'm all for
django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the
explanations I'm reading as I follow along with the progress of 'Django
Channels", just feels "less good" than solving this problem via Actor Model
Concurrency (championed by Erlang, Elixir, Stackless Python, etc).
There is an excellent actor model concurrency library for python that
already supports 'async django'. "Pulsar" (
http://pythonhosted.org/pulsar/index.html) and the documentation here
http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways
to use actor model concurrency to attack the issue of using Django with
asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels
sounds very similar to what they provide to work around the limitations of
django's blocking request processing cycle other parts sound much more
complicated, for instance the need for various back ends
in-memory/redis/etc.
I know Pulsar is currently Python 3.4 and newer only, but in the past it
supported 2.7, so anything we can learn from their concurrency approach can
work on Python 2.7 to maintain our backwards compatibility going forward.
You can see an example of Pulsar using python 2 if you look on the
benchmark page for their python powered 'redis clone' -
https://gist.github.com/lsbardel/8068579 - One of the benchmarks is
using pypy so the code definitely worked in a python 2.7 environment.
Would the work to integrate channels as part of Django improve the
ability to run Django in a completely async mode and enhance its
performance when run under Pulsar? or would it just be something that
needed to be turned off if we wanted to use something like Pulsar to power
our "async django". Is there anything to be gained from using a more actor
driven concurrency model or adopting any of the other methods used in their
'pulse' django app?
I suppose I'm just trying to work out how to compare using
Django-Channels with using Pulse from the Pulsar project.
On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
Post by Markus Holtermann
Post by Shai Berger
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
I don't agree; I think it's more "do you have enough workers to handle
this". Channels makes all workers serve all requests, so if you're doing
e.g. video encoding you can't segregate it from request handling. However,
you could have multiple channel backends and thus multiple worker pools to
achieve this.
Andrew
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uoBd6wC%2BcgXjQv%2BLZtGVqGRZLO2P6jeCyOokkC94XhjSQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Samuel Bishop
2015-12-27 07:53:55 UTC
Permalink
Once again I have to thank you for taking the time to keep replying. These
longer explanations are proving very helpful (at least to me) in
understanding Django Channels. :-)

I definitely understand the compatibility reasons, I'm also pleased to see
that signalling via a channel will be as simple as I had hoped, at least
simple as long as we put aside compatibility issues with Django's built in
signals. I'd still be pleased with only the ability to make my custom
signals asynchronous.

Digging into the implementation details, you've sparked my curiosity. For
some reason the simplicity of the having just send() and receive_many()
didn't fully dawn on me until now. Once I've made progress with my
decoupling, I think I'll have to try experiment with cleaving the major
parts of the entire request/response cycle into a chain of asynchronous
actors.
I'll likely test how hard it is to 'wrap' each part of the cycle in a way
that allows it to be called by synchronously or asynchronous code. I'd like
to avoid creating "yet another async django, that isnt really django
anymore", keep the existing synchronous code pathway seems like a good
start at a "useful experiment".

As for 'could you call the end result Django'... I think "Django" is
whatever we make of it ;-)
I don't see why we wont still call the combination of all the parts, the
ecosystem of compatible libraries, etc, "Django", regardless of how much we
might change between now and far future versions like 4.0 or 9.0.

Regards
Sam
Post by Andrew Godwin
Post by Samuel Bishop
Hi Andrew,
Thanks for the detailed response. I'm glad to hear you have tried a few
different ways to implement this, and having read through your reply, and
the channels docs again, I have more questions :-)
Your point about asynchronous code introducing needless complexity is
very well founded, its one of the reasons I like Actor Model Concurrency
over other concurrency models, it scales well computationally, and
*mentally*, as a programmer, I am trying break down my real world goals
into discrete steps of computations, comparisons, iterations and by
combining these steps, I construct my program. Actor Model Concurrency, is
no more complicated than saying "This step can be done without affecting
anything else thats going on", which makes the 'actor/task' much easier to
introduce to existing synchronous code than most other concurrency methods.
I agree - I'm a big fan of actors in general and it's usually what I turn
to to write complex simulations when I'm mucking around with game backends.
Post by Samuel Bishop
One thing that sticks with me as I think about the current channels
design, relates to the idea I've raised in another discussion (
https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ).
If I put my architect hat on and think about the future a bit, a 'more'
actor based design to channels might simplify later additional async work.
Since all the websocket stuff is new, I'll stick to the existing HTTP cycle
in this example. In version 1 a 'vanilla' Django task/actor would be a full
HTTP request response cycle, like how Django and WSGI currently work. Our
Django task is effectively our WSGI task, much like Pulsar Pulse. If all of
Django is contained inside our new task/worker/supervisor Actor layer, then
it becomes fairly trivial to do things like create separate request
handler, database query, and template rendering task/actors and chain their
calls. Since in the previously linked to discussion thread, I'm talking
about decoupling the ORM from the 'upper' layers of Django, its hard to not
also think what opportunities that would open up for async efforts. How do
you see the architecture of Django channels accommodating such improvements
in future? At the moment your focused on keeping things compatible and
avoiding the need for any of the existing synchronous Django code to
change, is this introducing any design tradeoffs that you are aware of with
regards to hypothetical future developments?
So, what you propose obviously isn't feasible in the short term - I'm
doing as much as I can to decouple Channels from "rewrite all of Django" -
but notably, channels is a subset of the actor model. In particular, you
could see consumers as stateless, single-action actors - but, this means
that any future abstraction would be able to build on that.
Post by Samuel Bishop
Django Channels feels like it contains a possible asynchronous
replacement for signals but is 'holding back', many cases where I can see
myself using a channel, I'm using them with a signal handler, can you
elaborate a bit on your thoughts here? I assume its a compatibility
decision, but is there more to it than just that?
It's entirely a compatibility decision - Channels needs to ship in a way
where code written for 1.9 will work the same, barring a few edge cases
where people dip down to raw WSGI.
You can entirely replace signals with custom channels, and I want to ship
example code that does so (it's two lines of code, pretty much), but they
work in a different way - for example, right now the pre_save signal can
actually prevent a save from occurring, whereas if it were on a channel
we'd have to then link the save consumer up to be run as a result of
pre_save, which starts down a path of compatibility nightmares.
Channels should be flexible enough that if you want to pursue the actor
pattern you can - the underlying library basically just has send() and
receive_many([channel_name, ...]) methods, so you can see how that could be
used to make different task-oriented processes rather than the generic
worker model that's going in now - it's just a case of keeping a major
addition like this as simple and backwards-compatible as possible.
Rewriting everything from scratch would be lovely, but unfortunately part
of maintaining a framework is not doing that.
Part of me, though, would be very interested in such a project being
undertaken to see how far it could go, though I'm not sure you could then
call the end result Django - at least, not without getting there
incrementally over many releases.
Andrew
Post by Samuel Bishop
Post by Andrew Godwin
Hi Samuel,
You'll notice if you look carefully that I avoid saying "async" almost
anywhere in the Channels announcements or documentation, and when I do,
it's qualified very carefully. Channels makes Django event-driven, but it
does not provide full "in-process" async like Twisted, asyncio, or
something like Pulsar.
Channels is actually kind of related to the actor model in some ways,
but it's simpler; imagine it as a pool of identical actors that are all
stateless, apart from the interface servers, which have the incoming socket
and thus the state.
The key thing is that Channels does nothing to make Django asynchronous.
All the Django code will remain synchronous (the ORM, templating, etc.),
and run in synchronous processes that just process one message after the
next. The only part of the system that will require an async framework is
the interface servers, as to be anywhere near efficient they need to serve
more than one socket at once; right now we're using Twisted and asyncio for
those, purely as they're incredibly mature and well-supported by the
community, but you could swap out anything there that achieved the same
goal (greenlets, for example, though I personally dislike their
implicitness).
Channels essentially takes the worker-prefork model that's been used in
webservers for years and applies it to a lower-level message concept rather
than HTTP requests (http requests are just one kind of message). This not
only has the advantage of a lot of previous work and research into
architecting and scaling these kinds of systems, it also means the actual
amount of work in core Django isn't crazy; it's more about providing nice
places to wrap up the worker loop and nice utilities around authentication
and session management.
If you read through the docs there's a bit more about this, but
basically, the fundamental design goal here is that Channels be no harder
to use than normal Django; indeed, it's almost exactly the same view
concept, but instead of a request you get a message, and the newer
developers I've interviewed about it seem to be comfortable with the
extension of the current abstraction straight away. I don't ever want
people to have to write asynchronous code if they don't want to; it's
currently so easy to shoot yourself in the foot in almost any Python async
solution that I don't think that makes for a good developer experience.
The "complexity" of the message backends you describe is because they're
network-transparent; there is indeed a simple, in-process one that uses
threads too, but the whole point of this design is that you run Django over
multiple processes on many machines, and in order to get something like
that you need a place to funnel messages through (another potential backend
design could use direct connection and process discovery, perhaps, or an
Erlang-like process spawn-and-track design with an overarching controller
process with access to many pre-selected machines).
The idea of making an asynchronous Django is tangential to making
Channels; this is more about changing the design abstraction of Django so
we can fit the concept in, and then providing a neat solution that drops
into the gap. There's nothing stopping someone making the worker processes
for Channels async-aware as well, so they can run multiple message
consumers at once, but that's such a complex and risk-fraught task that I
don't think it's worth it when we can get 90% of the benefits with a much
simpler and easier-to-design-and-understand system. If you want to see an
example of actual async Django, see Amber's talk from Django Under the Hood
this year, where she managed to get it working decently well under Twisted.
I expect Channels to smooth out Django's performance mostly as a
side-effect of the worker model providing demand-based load balancing by
design, but I doubt it will be a huge increase until people start using the
primitives it provides to offload tasks to other workers.
I hope that somewhat long-winded explanation helps - if not, let me know
what's still unclear and I'll try to tackle it. Clearing up what we're
trying to do and communicating it is important to me; I want to make sure
everyone knows so we can get good feedback on this stuff. For what it's
worth, I did consider the actor model along with several other kinds of
concurrency approaches (I've been prototyping bits of this for years); this
approach ended up being the nicest design and the one I have the most
confidence in that we can scale and shard to large work volumes.
Andrew
Post by Samuel Bishop
I'm uncertain how popular the suggestion will be but ... are "channels"
the right solution to our async/concurrency problems? ( I'm all for
django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the
explanations I'm reading as I follow along with the progress of 'Django
Channels", just feels "less good" than solving this problem via Actor Model
Concurrency (championed by Erlang, Elixir, Stackless Python, etc).
There is an excellent actor model concurrency library for python that
already supports 'async django'. "Pulsar" (
http://pythonhosted.org/pulsar/index.html) and the documentation here
http://pythonhosted.org/pulsar/apps/pulse.html shows two different
ways to use actor model concurrency to attack the issue of using Django
with asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels
sounds very similar to what they provide to work around the limitations of
django's blocking request processing cycle other parts sound much more
complicated, for instance the need for various back ends
in-memory/redis/etc.
I know Pulsar is currently Python 3.4 and newer only, but in the past
it supported 2.7, so anything we can learn from their concurrency approach
can work on Python 2.7 to maintain our backwards compatibility going
forward. You can see an example of Pulsar using python 2 if you look on the
benchmark page for their python powered 'redis clone' -
https://gist.github.com/lsbardel/8068579 - One of the benchmarks is
using pypy so the code definitely worked in a python 2.7 environment.
Would the work to integrate channels as part of Django improve the
ability to run Django in a completely async mode and enhance its
performance when run under Pulsar? or would it just be something that
needed to be turned off if we wanted to use something like Pulsar to power
our "async django". Is there anything to be gained from using a more actor
driven concurrency model or adopting any of the other methods used in their
'pulse' django app?
I suppose I'm just trying to work out how to compare using
Django-Channels with using Pulse from the Pulsar project.
On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
Post by Markus Holtermann
Post by Shai Berger
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
I don't agree; I think it's more "do you have enough workers to handle
this". Channels makes all workers serve all requests, so if you're doing
e.g. video encoding you can't segregate it from request handling. However,
you could have multiple channel backends and thus multiple worker pools to
achieve this.
Andrew
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cdf23837-6d82-418a-a034-aec2b6d2b1c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Andrew Godwin
2015-12-27 10:29:40 UTC
Permalink
Post by Samuel Bishop
Once again I have to thank you for taking the time to keep replying. These
longer explanations are proving very helpful (at least to me) in
understanding Django Channels. :-)
I definitely understand the compatibility reasons, I'm also pleased to see
that signalling via a channel will be as simple as I had hoped, at least
simple as long as we put aside compatibility issues with Django's built in
signals. I'd still be pleased with only the ability to make my custom
signals asynchronous.
Yes, the built-in signals have to remain as they are now; it is possible we
could hook something into the existing Signal code to provide an option to
run them via Channels, but I think the API to just make a new channel and
send events to it is actually easier and better anyway.

(I've never really liked the implicitness of channels, and the "come-from"
rather than "go-to" nature so you can't tell what code a signal invocation
might send without searching, whereas channel registrations are explicit in
a big dict)
Post by Samuel Bishop
Digging into the implementation details, you've sparked my curiosity. For
some reason the simplicity of the having just send() and receive_many()
didn't fully dawn on me until now. Once I've made progress with my
decoupling, I think I'll have to try experiment with cleaving the major
parts of the entire request/response cycle into a chain of asynchronous
actors.
Yup; in fact, I'm working on a spec, alongside development efforts, to
exactly lay out what the low-level API will be that everything runs on,
which is currently nicknamed "ASGI":
http://channels.readthedocs.org/en/latest/asgi.html

Django will run directly on top of that, with a small bit of code that
transforms send/receive_many into the documented consumer functions, but it
means there's flexibility to use the low-level API more directly in some
fashion if you want; since Channels is going to result in multiple projects
all cross-communicating already, I figured having a spec would be Very
Useful.
Post by Samuel Bishop
I'll likely test how hard it is to 'wrap' each part of the cycle in a way
that allows it to be called by synchronously or asynchronous code. I'd like
to avoid creating "yet another async django, that isnt really django
anymore", keep the existing synchronous code pathway seems like a good
start at a "useful experiment".
I am curious to see what you could achieve with more atomic breakdown of
Django view processing; things like template rendering can definitely be
done separately, while I'm less sure about the ORM (as you can't collect
results from multiple channels at once in Channels for performance reasons;
all you can do is listen for a single event on a single channel and react
to it before you get the next event).
Post by Samuel Bishop
As for 'could you call the end result Django'... I think "Django" is
whatever we make of it ;-)
I don't see why we wont still call the combination of all the parts, the
ecosystem of compatible libraries, etc, "Django", regardless of how much we
might change between now and far future versions like 4.0 or 9.0.
Well, "Django" is also something that's agreed upon by the community, the
Technical Board, and keeps to its promises of ease of use, backwards
compatibility, good documentation and so forth. Any radical change that
might end up happening would have to happen in slow steps over many
releases, and only with the buy-in of the community - I don't even fully
have that yet, and I don't expect to get it until Channels can show that it
works, is backwards-compatible, and solves issues without making anything
worse.

Andrew
Post by Samuel Bishop
Post by Andrew Godwin
Post by Samuel Bishop
Hi Andrew,
Thanks for the detailed response. I'm glad to hear you have tried a few
different ways to implement this, and having read through your reply, and
the channels docs again, I have more questions :-)
Your point about asynchronous code introducing needless complexity is
very well founded, its one of the reasons I like Actor Model Concurrency
over other concurrency models, it scales well computationally, and
*mentally*, as a programmer, I am trying break down my real world goals
into discrete steps of computations, comparisons, iterations and by
combining these steps, I construct my program. Actor Model Concurrency, is
no more complicated than saying "This step can be done without affecting
anything else thats going on", which makes the 'actor/task' much easier to
introduce to existing synchronous code than most other concurrency methods.
I agree - I'm a big fan of actors in general and it's usually what I turn
to to write complex simulations when I'm mucking around with game backends.
Post by Samuel Bishop
One thing that sticks with me as I think about the current channels
design, relates to the idea I've raised in another discussion (
https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ).
If I put my architect hat on and think about the future a bit, a 'more'
actor based design to channels might simplify later additional async work.
Since all the websocket stuff is new, I'll stick to the existing HTTP cycle
in this example. In version 1 a 'vanilla' Django task/actor would be a full
HTTP request response cycle, like how Django and WSGI currently work. Our
Django task is effectively our WSGI task, much like Pulsar Pulse. If all of
Django is contained inside our new task/worker/supervisor Actor layer, then
it becomes fairly trivial to do things like create separate request
handler, database query, and template rendering task/actors and chain their
calls. Since in the previously linked to discussion thread, I'm talking
about decoupling the ORM from the 'upper' layers of Django, its hard to not
also think what opportunities that would open up for async efforts. How do
you see the architecture of Django channels accommodating such improvements
in future? At the moment your focused on keeping things compatible and
avoiding the need for any of the existing synchronous Django code to
change, is this introducing any design tradeoffs that you are aware of with
regards to hypothetical future developments?
So, what you propose obviously isn't feasible in the short term - I'm
doing as much as I can to decouple Channels from "rewrite all of Django" -
but notably, channels is a subset of the actor model. In particular, you
could see consumers as stateless, single-action actors - but, this means
that any future abstraction would be able to build on that.
Post by Samuel Bishop
Django Channels feels like it contains a possible asynchronous
replacement for signals but is 'holding back', many cases where I can see
myself using a channel, I'm using them with a signal handler, can you
elaborate a bit on your thoughts here? I assume its a compatibility
decision, but is there more to it than just that?
It's entirely a compatibility decision - Channels needs to ship in a way
where code written for 1.9 will work the same, barring a few edge cases
where people dip down to raw WSGI.
You can entirely replace signals with custom channels, and I want to ship
example code that does so (it's two lines of code, pretty much), but they
work in a different way - for example, right now the pre_save signal can
actually prevent a save from occurring, whereas if it were on a channel
we'd have to then link the save consumer up to be run as a result of
pre_save, which starts down a path of compatibility nightmares.
Channels should be flexible enough that if you want to pursue the actor
pattern you can - the underlying library basically just has send() and
receive_many([channel_name, ...]) methods, so you can see how that could be
used to make different task-oriented processes rather than the generic
worker model that's going in now - it's just a case of keeping a major
addition like this as simple and backwards-compatible as possible.
Rewriting everything from scratch would be lovely, but unfortunately part
of maintaining a framework is not doing that.
Part of me, though, would be very interested in such a project being
undertaken to see how far it could go, though I'm not sure you could then
call the end result Django - at least, not without getting there
incrementally over many releases.
Andrew
Post by Samuel Bishop
Post by Andrew Godwin
Hi Samuel,
You'll notice if you look carefully that I avoid saying "async" almost
anywhere in the Channels announcements or documentation, and when I do,
it's qualified very carefully. Channels makes Django event-driven, but it
does not provide full "in-process" async like Twisted, asyncio, or
something like Pulsar.
Channels is actually kind of related to the actor model in some ways,
but it's simpler; imagine it as a pool of identical actors that are all
stateless, apart from the interface servers, which have the incoming socket
and thus the state.
The key thing is that Channels does nothing to make Django
asynchronous. All the Django code will remain synchronous (the ORM,
templating, etc.), and run in synchronous processes that just process one
message after the next. The only part of the system that will require an
async framework is the interface servers, as to be anywhere near efficient
they need to serve more than one socket at once; right now we're using
Twisted and asyncio for those, purely as they're incredibly mature and
well-supported by the community, but you could swap out anything there that
achieved the same goal (greenlets, for example, though I personally dislike
their implicitness).
Channels essentially takes the worker-prefork model that's been used in
webservers for years and applies it to a lower-level message concept rather
than HTTP requests (http requests are just one kind of message). This not
only has the advantage of a lot of previous work and research into
architecting and scaling these kinds of systems, it also means the actual
amount of work in core Django isn't crazy; it's more about providing nice
places to wrap up the worker loop and nice utilities around authentication
and session management.
If you read through the docs there's a bit more about this, but
basically, the fundamental design goal here is that Channels be no harder
to use than normal Django; indeed, it's almost exactly the same view
concept, but instead of a request you get a message, and the newer
developers I've interviewed about it seem to be comfortable with the
extension of the current abstraction straight away. I don't ever want
people to have to write asynchronous code if they don't want to; it's
currently so easy to shoot yourself in the foot in almost any Python async
solution that I don't think that makes for a good developer experience.
The "complexity" of the message backends you describe is because
they're network-transparent; there is indeed a simple, in-process one that
uses threads too, but the whole point of this design is that you run Django
over multiple processes on many machines, and in order to get something
like that you need a place to funnel messages through (another potential
backend design could use direct connection and process discovery, perhaps,
or an Erlang-like process spawn-and-track design with an overarching
controller process with access to many pre-selected machines).
The idea of making an asynchronous Django is tangential to making
Channels; this is more about changing the design abstraction of Django so
we can fit the concept in, and then providing a neat solution that drops
into the gap. There's nothing stopping someone making the worker processes
for Channels async-aware as well, so they can run multiple message
consumers at once, but that's such a complex and risk-fraught task that I
don't think it's worth it when we can get 90% of the benefits with a much
simpler and easier-to-design-and-understand system. If you want to see an
example of actual async Django, see Amber's talk from Django Under the Hood
this year, where she managed to get it working decently well under Twisted.
I expect Channels to smooth out Django's performance mostly as a
side-effect of the worker model providing demand-based load balancing by
design, but I doubt it will be a huge increase until people start using the
primitives it provides to offload tasks to other workers.
I hope that somewhat long-winded explanation helps - if not, let me
know what's still unclear and I'll try to tackle it. Clearing up what we're
trying to do and communicating it is important to me; I want to make sure
everyone knows so we can get good feedback on this stuff. For what it's
worth, I did consider the actor model along with several other kinds of
concurrency approaches (I've been prototyping bits of this for years); this
approach ended up being the nicest design and the one I have the most
confidence in that we can scale and shard to large work volumes.
Andrew
Post by Samuel Bishop
I'm uncertain how popular the suggestion will be but ... are
"channels" the right solution to our async/concurrency problems? ( I'm all
for django-channels the project and the work to solve these issues)
All the talk of channels, workers, queues and back ends in the
explanations I'm reading as I follow along with the progress of 'Django
Channels", just feels "less good" than solving this problem via Actor Model
Concurrency (championed by Erlang, Elixir, Stackless Python, etc).
There is an excellent actor model concurrency library for python that
already supports 'async django'. "Pulsar" (
http://pythonhosted.org/pulsar/index.html) and the documentation here
http://pythonhosted.org/pulsar/apps/pulse.html shows two different
ways to use actor model concurrency to attack the issue of using Django
with asynchronous web stuff like Websockets, etc.
Some of what's being described in the explanations of Django channels
sounds very similar to what they provide to work around the limitations of
django's blocking request processing cycle other parts sound much more
complicated, for instance the need for various back ends
in-memory/redis/etc.
I know Pulsar is currently Python 3.4 and newer only, but in the past
it supported 2.7, so anything we can learn from their concurrency approach
can work on Python 2.7 to maintain our backwards compatibility going
forward. You can see an example of Pulsar using python 2 if you look on the
benchmark page for their python powered 'redis clone' -
https://gist.github.com/lsbardel/8068579 - One of the benchmarks is
using pypy so the code definitely worked in a python 2.7 environment.
Would the work to integrate channels as part of Django improve the
ability to run Django in a completely async mode and enhance its
performance when run under Pulsar? or would it just be something that
needed to be turned off if we wanted to use something like Pulsar to power
our "async django". Is there anything to be gained from using a more actor
driven concurrency model or adopting any of the other methods used in their
'pulse' django app?
I suppose I'm just trying to work out how to compare using
Django-Channels with using Pulse from the Pulsar project.
On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <
Post by Markus Holtermann
Post by Shai Berger
If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.
For me it's more a question of "is a client directly involved".
I don't agree; I think it's more "do you have enough workers to
handle this". Channels makes all workers serve all requests, so if you're
doing e.g. video encoding you can't segregate it from request handling.
However, you could have multiple channel backends and thus multiple worker
pools to achieve this.
Andrew
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/cdf23837-6d82-418a-a034-aec2b6d2b1c2%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/cdf23837-6d82-418a-a034-aec2b6d2b1c2%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upkt4%3DnAo8JYgLVE0in8KnyZC3rJ0ayyyb-VrW8wnq%3D-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loading...