Explore integrating django-docker-box in some way?

[Tom Forbes]

Hello all,

I’ve been working on a docker-compose based alternative to django-box
(imaginatively named django-docker-box) over the last month and it finally
appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition
that is able to run a complete test matrix of every supported Python and DB
version. It’s as simple as docker-compose run sqlite. You can see a full
test run (excluding oracle) here:
https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any
potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the
Django project itself, maybe even as a replacement for django-box? I think
the setup is pretty quick compared to django-box and is more flexible in
terms of database version support as well as working with Oracle. I’d also
really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins
setup? I think it would be pretty neat to have parity between what runs on
the CI and what we can run locally and have any improvements shared between
both. Perhaps a full matrix run (which right now is 66 different
environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the
burden of running Jenkins by running the tests on a managed CI service like
Travis CI? We would likely still need Jenkins due to issues with Oracle and
running tests on Windows (unless https://github.com/django/django/pull/10259
works with Docker!), but we could offload some of the environments onto a
third party service. Travis gives large OS projects like Django increased
concurrency limits on their accounts so we could end up with pretty speedy
test runs. Also with docker-compose switching between CI services
(including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly
appreciated,

Tom

[Josh Smeaton]

I'm sorry I haven't had the time to review or contribute yet, but I think
it'll be a very useful project - especially for new contributors that might
have a little docker experience. The current vagrant solution is heavy,
does not work properly on windows and some linuxes, and isn't that easy to
maintain or deploy. I'd be in favour of adding the docker files directly to
django/django to minimise setup burden (DJANGO_PATH), and improving the
contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development
environments setup to effectively contribute - even using the docker-box
project which I understand quite well. Anything we can do to improve that
situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the
infrastructure team can comment more on that. I think that replacing the
ansible roles with a docker setup can have some definite improvements and
open up CI tasks to a larger pool of people (anyone that can edit docker
files), but it'd come with maintaining the host that runs docker (cleaning
up images, dealing with disk space issues, etc).

Post by Tom Forbes
Hello all,
I’ve been working on a docker-compose based alternative to django-box
(imaginatively named django-docker-box) over the last month and it
finally appears to be mostly complete.
For reference the tool is just a Dockerfile and a docker-compose
definition that is able to run a complete test matrix of every supported
Python and DB version. It’s as simple as docker-compose run sqlite. You
https://travis-ci.com/orf/django-docker-box/builds/90167436
Florian suggested I create a thread here to gather feedback and discuss
Firstly I’d like to know if there is any support for moving this under the
Django project itself, maybe even as a replacement for django-box? I think
the setup is pretty quick compared to django-box and is more flexible in
terms of database version support as well as working with Oracle. I’d also
really like some help improving Oracle support if anyone has the time!
Secondly is there any support for integrating this with our current
Jenkins setup? I think it would be pretty neat to have parity between what
runs on the CI and what we can run locally and have any improvements shared
between both. Perhaps a full matrix run (which right now is 66 different
environments) is out of the question but a smaller subset could be good?
Thirdly, and this is a bit wild, but what about using this to reduce the
burden of running Jenkins by running the tests on a managed CI service like
Travis CI? We would likely still need Jenkins due to issues with Oracle and
running tests on Windows (unless
https://github.com/django/django/pull/10259 works with Docker!), but we
could offload some of the environments onto a third party service. Travis
gives large OS projects like Django increased concurrency limits on their
accounts so we could end up with pretty speedy test runs. Also with
docker-compose switching between CI services (including Jenkins) would be
very simple.
The repo is here: https://github.com/orf/django-docker-box.
Any feedback on these points or the project itself would be greatly
appreciated,
Tom

[Tom Forbes]

Thank you for the reply Josh. I didn’t anticipate any suggestions for
including this inside core but off the back of your suggestion I’ve made a
ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django -
it’s four or five new files at most. Hopefully this should improve the
experience at sprints, however the current Dockerfile weighs in at 650+mb
so the problem may switch from ‘it is hard to set up an environment’ to ‘it
is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton (***@gmail.com) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think
it'll be a very useful project - especially for new contributors that might
have a little docker experience. The current vagrant solution is heavy,
does not work properly on windows and some linuxes, and isn't that easy to
maintain or deploy. I'd be in favour of adding the docker files directly to
django/django to minimise setup burden (DJANGO_PATH), and improving the
contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development
environments setup to effectively contribute - even using the docker-box
project which I understand quite well. Anything we can do to improve that
situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the
infrastructure team can comment more on that. I think that replacing the
ansible roles with a docker setup can have some definite improvements and
open up CI tasks to a larger pool of people (anyone that can edit docker
files), but it'd come with maintaining the host that runs docker (cleaning
up images, dealing with disk space issues, etc).

Post by Tom Forbes
Hello all,
I’ve been working on a docker-compose based alternative to django-box
(imaginatively named django-docker-box) over the last month and it
finally appears to be mostly complete.
For reference the tool is just a Dockerfile and a docker-compose
definition that is able to run a complete test matrix of every supported
Python and DB version. It’s as simple as docker-compose run sqlite. You
https://travis-ci.com/orf/django-docker-box/builds/90167436
Florian suggested I create a thread here to gather feedback and discuss
Firstly I’d like to know if there is any support for moving this under the
Django project itself, maybe even as a replacement for django-box? I think
the setup is pretty quick compared to django-box and is more flexible in
terms of database version support as well as working with Oracle. I’d also
really like some help improving Oracle support if anyone has the time!
Secondly is there any support for integrating this with our current
Jenkins setup? I think it would be pretty neat to have parity between what
runs on the CI and what we can run locally and have any improvements shared
between both. Perhaps a full matrix run (which right now is 66 different
environments) is out of the question but a smaller subset could be good?
Thirdly, and this is a bit wild, but what about using this to reduce the
burden of running Jenkins by running the tests on a managed CI service like
Travis CI? We would likely still need Jenkins due to issues with Oracle and
running tests on Windows (unless https://github.com/django/
django/pull/10259 works with Docker!), but we could offload some of the
environments onto a third party service. Travis gives large OS projects
like Django increased concurrency limits on their accounts so we could end
up with pretty speedy test runs. Also with docker-compose switching between
CI services (including Jenkins) would be very simple.
The repo is here: https://github.com/orf/django-docker-box.
Any feedback on these points or the project itself would be greatly
appreciated,
Tom
--

You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Josh Smeaton]

Size of the image could definitely be a concern, especially at sprints
where wifi speeds aren't always optimal. The django-box image is
significantly larger though so it'd still be a net win. There are also
optimisations that can be made to the image for reducing size over time, so
I'd fully expect it to come down. I've spent a little bit of time trying to
optimise a $work$ python docker file, I'll provide what I've got as an
issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step
forward. I'd also like to hear from the infrastructure team what their
thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this
list of names is from memory).

Post by Tom Forbes
Thank you for the reply Josh. I didn’t anticipate any suggestions for
including this inside core but off the back of your suggestion I’ve made a
ticket here: https://code.djangoproject.com/ticket/30010.
I don’t think it should be complex at all to include this inside Django -
it’s four or five new files at most. Hopefully this should improve the
experience at sprints, however the current Dockerfile weighs in at 650+mb
so the problem may switch from ‘it is hard to set up an environment’ to ‘it
is hard to download one’!
I'm sorry I haven't had the time to review or contribute yet, but I think
it'll be a very useful project - especially for new contributors that might
have a little docker experience. The current vagrant solution is heavy,
does not work properly on windows and some linuxes, and isn't that easy to
maintain or deploy. I'd be in favour of adding the docker files directly to
django/django to minimise setup burden (DJANGO_PATH), and improving the
contributing docs to show users how to test using docker.
One of the hardest things I found at sprints was getting development
environments setup to effectively contribute - even using the docker-box
project which I understand quite well. Anything we can do to improve that
situation will be very beneficial.
I have fewer opinions about the official CI story, hopefully some of the
infrastructure team can comment more on that. I think that replacing the
ansible roles with a docker setup can have some definite improvements and
open up CI tasks to a larger pool of people (anyone that can edit docker
files), but it'd come with maintaining the host that runs docker (cleaning
up images, dealing with disk space issues, etc).

Post by Tom Forbes
Hello all,
I’ve been working on a docker-compose based alternative to django-box
(imaginatively named django-docker-box) over the last month and it
finally appears to be mostly complete.
For reference the tool is just a Dockerfile and a docker-compose
definition that is able to run a complete test matrix of every supported
Python and DB version. It’s as simple as docker-compose run sqlite. You
https://travis-ci.com/orf/django-docker-box/builds/90167436
Florian suggested I create a thread here to gather feedback and discuss
Firstly I’d like to know if there is any support for moving this under
the Django project itself, maybe even as a replacement for django-box? I
think the setup is pretty quick compared to django-box and is more flexible
in terms of database version support as well as working with Oracle. I’d
also really like some help improving Oracle support if anyone has the time!
Secondly is there any support for integrating this with our current
Jenkins setup? I think it would be pretty neat to have parity between what
runs on the CI and what we can run locally and have any improvements shared
between both. Perhaps a full matrix run (which right now is 66 different
environments) is out of the question but a smaller subset could be good?
Thirdly, and this is a bit wild, but what about using this to reduce the
burden of running Jenkins by running the tests on a managed CI service like
Travis CI? We would likely still need Jenkins due to issues with Oracle and
running tests on Windows (unless
https://github.com/django/django/pull/10259 works with Docker!), but we
could offload some of the environments onto a third party service. Travis
gives large OS projects like Django increased concurrency limits on their
accounts so we could end up with pretty speedy test runs. Also with
docker-compose switching between CI services (including Jenkins) would be
very simple.
The repo is here: https://github.com/orf/django-docker-box.
Any feedback on these points or the project itself would be greatly
appreciated,
Tom
--

You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Tom Forbes]

To have this completely working at sprints without having everyone building
their own local images we would need to have the Jenkins server use docker
in some capacity. This would also require an official django account on
Docker hub.

The pattern I’m using right now is that on every build we pull the
django-ci:latest image (from my personal account). Docker uses this image
as a cache automatically (preventing rebuilds). On any successful master
build we push the new image to docker-hub, so subsequent builds can utilise
it.

Then anyone wanting to speed up their bootstrapping can do docker-compose
pull and automatically have the latest image available for running right
away. We can make this smaller, for sure, but we can also suggest people
download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching
docker mirror (docker run -p 5000:5000 registry). Organizers could run this
at large events and configuring docker to use a local mirror on the network
is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton (***@gmail.com) wrote:

Size of the image could definitely be a concern, especially at sprints
where wifi speeds aren't always optimal. The django-box image is
significantly larger though so it'd still be a net win. There are also
optimisations that can be made to the image for reducing size over time, so
I'd fully expect it to come down. I've spent a little bit of time trying to
optimise a $work$ python docker file, I'll provide what I've got as an
issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step
forward. I'd also like to hear from the infrastructure team what their
thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this
list of names is from memory).

Post by Tom Forbes
Thank you for the reply Josh. I didn’t anticipate any suggestions for
including this inside core but off the back of your suggestion I’ve made a
ticket here: https://code.djangoproject.com/ticket/30010.
I don’t think it should be complex at all to include this inside Django -
it’s four or five new files at most. Hopefully this should improve the
experience at sprints, however the current Dockerfile weighs in at 650+mb
so the problem may switch from ‘it is hard to set up an environment’ to ‘it
is hard to download one’!
I'm sorry I haven't had the time to review or contribute yet, but I think
it'll be a very useful project - especially for new contributors that might
have a little docker experience. The current vagrant solution is heavy,
does not work properly on windows and some linuxes, and isn't that easy to
maintain or deploy. I'd be in favour of adding the docker files directly to
django/django to minimise setup burden (DJANGO_PATH), and improving the
contributing docs to show users how to test using docker.
One of the hardest things I found at sprints was getting development
environments setup to effectively contribute - even using the docker-box
project which I understand quite well. Anything we can do to improve that
situation will be very beneficial.
I have fewer opinions about the official CI story, hopefully some of the
infrastructure team can comment more on that. I think that replacing the
ansible roles with a docker setup can have some definite improvements and
open up CI tasks to a larger pool of people (anyone that can edit docker
files), but it'd come with maintaining the host that runs docker (cleaning
up images, dealing with disk space issues, etc).

Post by Tom Forbes
Hello all,
I’ve been working on a docker-compose based alternative to django-box
(imaginatively named django-docker-box) over the last month and it
finally appears to be mostly complete.
For reference the tool is just a Dockerfile and a docker-compose
definition that is able to run a complete test matrix of every supported
Python and DB version. It’s as simple as docker-compose run sqlite. You
https://travis-ci.com/orf/django-docker-box/builds/90167436
Florian suggested I create a thread here to gather feedback and discuss
Firstly I’d like to know if there is any support for moving this under
the Django project itself, maybe even as a replacement for django-box? I
think the setup is pretty quick compared to django-box and is more flexible
in terms of database version support as well as working with Oracle. I’d
also really like some help improving Oracle support if anyone has the time!
Secondly is there any support for integrating this with our current
Jenkins setup? I think it would be pretty neat to have parity between what
runs on the CI and what we can run locally and have any improvements shared
between both. Perhaps a full matrix run (which right now is 66 different
environments) is out of the question but a smaller subset could be good?
Thirdly, and this is a bit wild, but what about using this to reduce the
burden of running Jenkins by running the tests on a managed CI service like
Travis CI? We would likely still need Jenkins due to issues with Oracle and
running tests on Windows (unless https://github.com/django/
django/pull/10259 works with Docker!), but we could offload some of the
environments onto a third party service. Travis gives large OS projects
like Django increased concurrency limits on their accounts so we could end
up with pretty speedy test runs. Also with docker-compose switching between
CI services (including Jenkins) would be very simple.
The repo is here: https://github.com/orf/django-docker-box.
Any feedback on these points or the project itself would be greatly
appreciated,
Tom
--

You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/074f68c9-2199-4128-a37a-
bfc1852f4806%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--

You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+***@googlegroups.com.
To post to this group, send email to django-***@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/64a2874a-01d1-4ffd-acaa-a811ee5de5ee%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.