A proposal for a new auto-reloader in Django

[q***@gmail.com]

Hi, I'm David Ma, a first-year Science student from the University of
British Columbia. I'm a enthusiastic Django developer and have four years
Python programming experience. I've read through the posts about replacing
the current autoreloader and would like to work on this task during GSoC. I
paraphrase the discussions as follows:

Currently, the default autoreloader contains a lot of problems, most of
which are caused by the inefficient methods to look for file changes
(#27685). Notably, it fails to work with many python libraries including
Jinja (#25624). In addition, if a syntax error is in the settings file, the
autoreloader would stop working. We want to get rid of this by tracking the
status of an entire folder and using subprocesses to reload the development
server.

On going discussions have deemed that the best way to solve this is to
replace the current autoreloader with a new one implemented with Watchman
(https://groups.google.com/forum/#!topic/django-developers/voXNIDdDcpU/discussion).
Since inotify is a low-level library and cannot be applied to platforms
including Windows and Macintosh OS, the best library to replace Watchdog is
Watchman, a library that supports both of these three platforms (for
Windows it's still in the alpha stage.) It runs as a standalone process and
has a python wrapper library that is incompatible with Python 3. Therefore,
one of the goals is to figure out how to interact with this library. There
are also complaints on Django not tracking non-python files or tracking
files that it is not supposed to watch. For the new version, we'd expect it
to work with cached templates (#25791) and be customizable.


So my goals would be:


1.

Replace the old auto-reloader with the Watchman version
2.

Make sure that the new auto-reloader can work with Python3 and on other
platforms
3.

Make it compatible with other Python libraries
4.

Allow Django users to choose the files that should be watched
5.

Integrate the Watchman package and its Python wrapper into DjangoI


For Phase 1, I will write a prototype that can replace the current
autoreloader. In the second phase, I'll come up with another one capable of
handling cached files and excluding certain types of files based on the
configuration, and the final version will be a cross-platform autoreloader
that works with most of the Python libraries. Here's my development
schedule:

- May 30 – June 8: Surveying the functions implemented in
autoreloader.py, researching the dependency of the functions included, and
constructing the development environment.
- June 8 – June 17: Play with the Watchman library; design a way to
interact with Watchman (probably as a standalone process); write wrappers
for Watchman. Make a list of files that is not monitored by default (e.g.
sqlite files or files under the media directory) and make sure it works
with settings.py.
- June 17 – June 25: Implement the auto-reloader from scratch; make sure
the new reloader can work with Django properly (particularly can survive a
syntax error and would not be affected by cached files.)
- July 26 – June 30: Phase 1 evaluations
- July 1 – July 8: Add appropriate benchmarks and check whether if the
new implementation meets the expectation. Test whether if the new
implementation can detect the addition of a file in the trivial cases and
work with applications like git.
- July 8 – July 15: Figure out how to monitor cached files in Django;
add an option that enables Watchman to ignore all cached files. Check
whether if this solution works with cached templates.
- July 15 – July 24: Designate options to control the files monitored by
Watchman with a standard include / exclude mechanism; write a function to
read the relevant portion in the configuration file.
- July 24 – July 28: Phase 2 evaluations
- August 2 – August 15: Test compatibility with other libraries
including Jinja2 and Jython; revise the code if necessary. Test the
autoreloader on Windows and Macintosh OS.
- August 15 – August 21: Integrate the Watchman library and its
wrappers into Django (might need to submit pull requests to the pywatchman
project.)
- August 21 – August 29: Final evaluations


Thanks for reading my proposal. If you have any questions or concerns,
please do not hesitate to comment. Any advice and suggestions would be
greatly appreciated!

[Brice PARENT]

Hi David,
I think it's a good idea to improve what we have now, mostly for the
problems you described.
Don't you think it would be a better idea to make a reloader which
doesn't depend on Django at all? You could make a library working with a
settings file, and create a base settings file for Django projects.
You could define whatever should be called to start and restart the
server, and be able to define file types and folders that should be seen
as changes that require a restart or not. It could then be used easily
with any kind of Python project, and even allow specific behaviours,
like launching a server reload when a new plugin is uploaded to a
specific folder, or when one of them is updated.
Every Python framework could then embed their own base settings file,
which would allow this project to touch a wider audience.

- Brice

[q***@gmail.com]

Hi Brice,
Thanks for the suggestion! I'm not sure whether if I can successfully
implement a library like you've mentioned, so for now I might stick with
the Watch* libraries available. I really love the goals you've listed and
would add these to my proposal.

David Ma

Post by Brice PARENT
Hi David,
I think it's a good idea to improve what we have now, mostly for the
problems you described.
Don't you think it would be a better idea to make a reloader which
doesn't depend on Django at all? You could make a library working with a
settings file, and create a base settings file for Django projects.
You could define whatever should be called to start and restart the
server, and be able to define file types and folders that should be seen
as changes that require a restart or not. It could then be used easily
with any kind of Python project, and even allow specific behaviours,
like launching a server reload when a new plugin is uploaded to a
specific folder, or when one of them is updated.
Every Python framework could then embed their own base settings file,
which would allow this project to touch a wider audience.
- Brice

[Aymeric Augustin]

Hello,

the best library to replace Watchdog is Watchman, a library that supports both of these three platforms (for Windows it's still in the alpha stage.)

Django currently doesn't use Watchdog. Watchdog is a cross-platform Python library that can be used for implementing an autoreloader. It's a plausible alternative to Watchman, likely easier to integrate, but Watchman has some fancy features that I'd love to have available -- like waiting for git to finish its operations before notifying a change.

It runs as a standalone process and has a python wrapper library that is incompatible with Python 3. Therefore, one of the goals is to figure out how to interact with this library.

In order to be selected for this project, you have to convince us that you can solve this problem. At a minimum you must prove that you understand where the difficulties lie and suggest some possible approaches to tackle them.

• Replace the old auto-reloader with the Watchman version

Since installing watchman isn't trivial, you need to keep a pure-Python alternative so that beginners get an acceptable experience out of the box. This alternative may be the current "poll every second approach" or could be backed by watchdog, assuming it's pip-installable without complicated dependencies (e.g. a C compiler and development headers for a bunch of libraries) on all operating systems.

• Make sure that the new auto-reloader can work with Python3 and on other platforms

Did you mean "work with Python 2" ? In any case, this isn't necessary, as you'd be working off the master branch which no longer supports Python 2.

• Make it compatible with other Python libraries
• Allow Django users to choose the files that should be watched
• Integrate the Watchman package and its Python wrapper into Django

Watchman should be installed separately by developers who wish to use it. Django should use it when it's available (and there should be a flag to disable that if for any reason someone has watchman installed systemwide but doesn't want to use it).

I hope this helps!
--
Aymeric.

[Carl Meyer]

Anyone working on this project should at least be aware of
https://github.com/Pylons/hupper (based on work David Glick and I
originally did in https://github.com/carljm/wsgiwatcher), which aims to
be a framework-agnostic solution to this problem for any Python web
project. Docs at http://docs.pylonsproject.org/projects/hupper/en/latest/

Carl

[q***@gmail.com]

Hi Carl,
Thanks for mentioning this awesome project! I saw it in one of the
discussions but did not take a close look. I'll definitely check this out
and try to integrate wsgiwatcher/watcher.py into Django.

Post by Carl Meyer
Anyone working on this project should at least be aware of
https://github.com/Pylons/hupper
<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FPylons%2Fhupper&sa=D&sntz=1&usg=AFQjCNGVwtqvdo53UFfK80kaQ1qxL7ST8Q>
(based on work David Glick and I
originally did in https://github.com/carljm/wsgiwatcher
<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fcarljm%2Fwsgiwatcher&sa=D&sntz=1&usg=AFQjCNGqLfsmxsp37Lng7_d_DaVzja9c6Q>),
which aims to
be a framework-agnostic solution to this problem for any Python web
project. Docs at http://docs.pylonsproject.org/projects/hupper/en/latest/
Carl

[Melvyn Sopacua]

Post by q***@gmail.com
Hi Carl,
Thanks for mentioning this awesome project! I saw it in one of the
discussions but did not take a close look. I'll definitely check this
out and try to integrate wsgiwatcher/watcher.py into Django.

Post by Carl Meyer
Anyone working on this project should at least be aware of
https://github.com/Pylons/hupper
<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FPylons%2Fhu
pper&sa=D&sntz=1&usg=AFQjCNGVwtqvdo53UFfK80kaQ1qxL7ST8Q> (based

on

Post by q***@gmail.com

Post by Carl Meyer
work David Glick and I
originally did in https://github.com/carljm/wsgiwatcher
<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fcarljm%2Fws
giwatcher&sa=D&sntz=1&usg=AFQjCNGqLfsmxsp37Lng7_d_DaVzja9c6Q>),
which aims to
be a framework-agnostic solution to this problem for any Python web
project. Docs at
http://docs.pylonsproject.org/projects/hupper/en/latest/
Carl

I was going to suggest the same, because you already get watchdog support for free
and can probably build on that integration to pick up watchman. Also, it's used in
devpi[1], so it gets some scruteny from there (and devpi is awesome to distribute
internal or augmented Django apps amongst projects).
--
Melvyn Sopacua

--------
[1] http://doc.devpi.net/

[q***@gmail.com]

Hi Carl,
I don't quite understand why get_module_paths() in your wsgiwatcher
project is returning a list of python module paths. I thought it would
return the directory that needs to be monitored. Could you please tell me
how this part works? Thanks.

David Ma

Post by Carl Meyer
Anyone working on this project should at least be aware of
https://github.com/Pylons/hupper (based on work David Glick and I
originally did in https://github.com/carljm/wsgiwatcher), which aims to
be a framework-agnostic solution to this problem for any Python web
project. Docs at http://docs.pylonsproject.org/projects/hupper/en/latest/
<http://www.google.com/url?q=http%3A%2F%2Fdocs.pylonsproject.org%2Fprojects%2Fhupper%2Fen%2Flatest%2F&sa=D&sntz=1&usg=AFQjCNHhMlzZ6K6HLWsxUxOPanWzxLeMlg>
Carl

[Carl Meyer]

Hi David,

I wouldn't bother with the wsgiwatcher repo; it's proof-of-concept code
from one sprint's worth of hacking, not used by anyone. Look at hupper
instead, it is based on wsgiwatcher, but much changed, and actually used
in production.

Carl

Post by q***@gmail.com
Hi Carl,
I don't quite understand why get_module_paths() in your wsgiwatcher
project is returning a list of python module paths. I thought it would
return the directory that needs to be monitored. Could you please tell
me how this part works? Thanks.
David Ma
Anyone working on this project should at least be aware of
https://github.com/Pylons/hupper <https://github.com/Pylons/hupper>
(based on work David Glick and I
originally did in https://github.com/carljm/wsgiwatcher
<https://github.com/carljm/wsgiwatcher>), which aims to
be a framework-agnostic solution to this problem for any Python web
project. Docs at
http://docs.pylonsproject.org/projects/hupper/en/latest/
<http://www.google.com/url?q=http%3A%2F%2Fdocs.pylonsproject.org%2Fprojects%2Fhupper%2Fen%2Flatest%2F&sa=D&sntz=1&usg=AFQjCNHhMlzZ6K6HLWsxUxOPanWzxLeMlg>
Carl
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/4fac9a8a-231d-41c5-a5fb-6eaeb2bd148f%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/4fac9a8a-231d-41c5-a5fb-6eaeb2bd148f%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

[q***@gmail.com]

Hi Aymeric,

Thanks for this detailed and informative response! I will try to figure
out the difficulties of integrating Watchman during the weekend. For the
pure-Python solution, I might implement a standalone autoreloader based on
Carl's work and replace the current one. Does this look good to you?

David Ma

Post by Aymeric Augustin
Hello,

the best library to replace Watchdog is Watchman, a library that

supports both of these three platforms (for Windows it's still in the alpha
stage.)
Django currently doesn't use Watchdog. Watchdog is a cross-platform Python
library that can be used for implementing an autoreloader. It's a plausible
alternative to Watchman, likely easier to integrate, but Watchman has some
fancy features that I'd love to have available -- like waiting for git to
finish its operations before notifying a change.

It runs as a standalone process and has a python wrapper library that is

incompatible with Python 3. Therefore, one of the goals is to figure out
how to interact with this library.
In order to be selected for this project, you have to convince us that you
can solve this problem. At a minimum you must prove that you understand
where the difficulties lie and suggest some possible approaches to tackle
them.

• Replace the old auto-reloader with the Watchman version

Since installing watchman isn't trivial, you need to keep a pure-Python
alternative so that beginners get an acceptable experience out of the box.
This alternative may be the current "poll every second approach" or could
be backed by watchdog, assuming it's pip-installable without complicated
dependencies (e.g. a C compiler and development headers for a bunch of
libraries) on all operating systems.

• Make sure that the new auto-reloader can work with Python3 and

on other platforms
Did you mean "work with Python 2" ? In any case, this isn't necessary, as
you'd be working off the master branch which no longer supports Python 2.

• Make it compatible with other Python libraries
• Allow Django users to choose the files that should be watched
• Integrate the Watchman package and its Python wrapper into

Django
Watchman should be installed separately by developers who wish to use it.
Django should use it when it's available (and there should be a flag to
disable that if for any reason someone has watchman installed systemwide
but doesn't want to use it).
I hope this helps!
--
Aymeric.

[Aymeric Augustin]

Hello,

For the pure-Python solution, I might implement a standalone autoreloader based on Carl's work and replace the current one. Does this look good to you?

Perhaps.

It depends on the details (features, installation requirements, cross platform support, etc.).

It's up to you to argument in your proposal why this would be a good solution.

Best regards,
--
Aymeric.

[q***@gmail.com]

Hi Aymeric,
I feel like using Watchman might be the easiest solution. It keeps
triggering the command once an event occurs, so autoreload.py does not need
to restart the server in case of a syntax error. Pywatchman does not work
on Python 3 and has some dependency problems, so I'm planning to use the
command line interface of the Watchman library. Watchdog only triggers the
given command once, so it cannot survive a syntax error and requires the
auto-reloader to deal with multi-threading, the most challenging part in
this project (likely to enter the same problem described in #2330
<http://code.djangoproject.com/ticket/2330>.) Therefore, I'd start with
Watchman and see how to implement this with Watchdog later. How do you feel
about this plan?

Cheers,
David Ma

Post by Aymeric Augustin
Hello,
For the pure-Python solution, I might implement a standalone autoreloader
based on Carl's work and replace the current one. Does this look good to
you?
Perhaps.
It depends on the details (features, installation requirements, cross
platform support, etc.).
It's up to you to argument in your proposal why this would be a good solution.
Best regards,
--
Aymeric.

[Aymeric Augustin]

Hello David,

This reasoning makes sense.

You say that watchman "keeps triggering the command once an event occurs". This is correct; as a consequence "the command" must stop the currently running instance of the development server and start a new one. This requires overhauling significantly the current implementation of the development server.

You need to understand the two-process structure of the current autoreloader and the workarounds for allowing runserver to start with invalid settings to get a good grasp on the requirements for a good developer experience. I hope we can do away with these workarounds with a better architecture.

Best regards,
--
Aymeric.

Post by q***@gmail.com
Hi Aymeric,
I feel like using Watchman might be the easiest solution. It keeps triggering the command once an event occurs, so autoreload.py does not need to restart the server in case of a syntax error. Pywatchman does not work on Python 3 and has some dependency problems, so I'm planning to use the command line interface of the Watchman library. Watchdog only triggers the given command once, so it cannot survive a syntax error and requires the auto-reloader to deal with multi-threading, the most challenging part in this project (likely to enter the same problem described in #2330 <http://code.djangoproject.com/ticket/2330>.) Therefore, I'd start with Watchman and see how to implement this with Watchdog later. How do you feel about this plan?
Cheers,
David Ma
Hello,

For the pure-Python solution, I might implement a standalone autoreloader based on Carl's work and replace the current one. Does this look good to you?

Perhaps.
It depends on the details (features, installation requirements, cross platform support, etc.).
It's up to you to argument in your proposal why this would be a good solution.
Best regards,
--
Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
Visit this group at https://groups.google.com/group/django-developers <https://groups.google.com/group/django-developers>.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3c284301-15e9-44d9-9b7c-2d866b7f7d13%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/3c284301-15e9-44d9-9b7c-2d866b7f7d13%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.