Last time, we started trying to deploy a Django app to a Raspberry Pi. I came up with the following steps, and did some of them in the last post:

1. Get our python code onto the Pi.
2. Get the dependencies installed on the Pi.
3. Configure apache webserver to forward requests to django.
4. Set up a database
5. Probably other things I haven’t thought of yet.

1 and 2 took a little longer than I thought, and they are covered in the previous post. This post details my attempts at step 3. It turns out that I only thought I was done with step 2, but we’ll learn about that as we go.

I mostly followed this tutorial, with the difference that I used rsync to copy over an existing Django app, rather than just initiating one on the Pi. I also had more trouble getting Python working, because I have some dependencies that need a newer Python version than ships on the Raspberry Pi, and because I have some non-PyPi dependencies that need to download over https. I also had an existing project running on the apache2 webserver on my Raspberry Pi, and needed to turn it down to free up port 80, which is the usual port for HTTP. I got all that done, but now when I visit my Raspberry Pi in the browser, I see the very boring 404 page:

Not Found

The requested URL was not found on this server.
Apache/2.4.38 (Raspbian) Server at 192.168.0.105 Port 80

So this leaves us squarely in “Step 3. Configure apache webserver to forward requests to django.” And we have a genuine mystery on our hands, since I have no idea what’s causing the 404.

First, I notice that this isn’t a Django error page, it’s an apache2 error page. That makes me think that something is configured wrong such that the apache2 webserver doesn’t know about the routes that Django is serving, or maybe just can’t find the Django app at all.

To try to fix this, I’m going to take some time trying to understand the basics of configuring the apache2 webserver and getting it to talk to Django at all, and then maybe I’ll be able to figure it out.

Here are some initial hypotheses:

1. Apache is trying to start using the wrong python. We had to install a new python interpreter to get the Django app to install its dependencies, and so it seems possible that apache doesn’t know about this interpreter, or about it’s virtual environment, and so can’t really start the server process.
2. It’s possible I just missed a step, and I need to tell the django app to receive requests.
3. It’s possible I’m missing pieces of Apache’s config, and it doesn’t know I want it to forward requests to Django.

I’ve never tried to configure this setup before, so any of these could be happening. I feel like 2 and 3, or a combination, is likeliest. I thought there would be a setting that said something like “Dear Apache2 webserver, when you get a request on port 80, use this interpreter to invoke this python script.”

So let’s try to learn a bit about that. Here’s a super useful command I’ve been using to see what went wrong on the server:

$ tail -f /var/log/apache2/*.log

I like this command because the *.log on the end keeps me from having to figure out log rotation, or naming, or whether access.log or error.log has what I want.

With that command, I’m seeing errors in the log that look like:

[Sat Jun 17 19:39:25.378551 2023] [wsgi:error] [pid 21900] [client 192.168.0.34:58180] \
Target WSGI script not found or unable to stat: /home/pi/pidjango/pidjango

Fixing the WSGIScriptAlias path in the apache2 config worked. Now I get a 500. Time for a celebration dance! I got to a better error message. Let’s see what the logs say now!

Now the error looks like this:

 Traceback (most recent call last):
   File "/home/pi/pidjango/library_django/wsgi.py", line 12, in <module>
     from django.core.wsgi import get_wsgi_application
 ModuleNotFoundError: No module named 'django'

That makes me think there’s a problem with the virtual environment. In other words, when apache2 tries to invoke the django process, there’s no dependencies loaded. An experiment I should have run at this point but didn’t think of, would be to run python manage.py runserver directly on the Pi from inside the virtual environment. If that had worked, it would prove that the virtual environment was correctly configured. As it was, I just tentatively reached that conclusion by googling error messages.

Here are some of the things I found that made me suspect a different problem:

https://github.com/docker-library/python/issues/559#issuecomment-730736731 says:

So I guess the other way to put this is that if you want libapache2-mod-wsgi-py3 to work in this image and use our Python instead of Debian’s, you’re going to have to compile mod-wsgi from source against our Python plus Debian’s httpd headers (something that’s likely to be somewhat involved, but to what degree I couldn’t really wager a guess since I’m unfortunately totally unfamiliar with mod-wsgi).

And an old StackOverflow answer, which suggested I check how apache2’s mod_wsgi was linked:

 $ ldd /usr/lib/apache2/modules/mod_wsgi.so
	linux-vdso.so.1 (0x7eaec000)
	libpython3.7m.so.1.0 => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so.1.0 (0x76a69000)
	... snip ...

This is concerning. Last time, I fixed dependencies that were too new for the system python by installing a newer python alongside the existing one. But the ldd command above makes me think that apache2 knows that it’s supposed to work with python 3.7, and not a newer python. I think maybe I need to upgrade apache2 to get it to link with newer python.

Maybe we’ll get lucky and I can upgrade this easily?

$ sudo apt-get install apache2 libapache2-mod-wsgi-py3
Reading package lists... Done
Building dependency tree
Reading state information... Done
apache2 is already the newest version (2.4.38-3+deb10u10).
libapache2-mod-wsgi-py3 is already the newest version (4.6.5-1+deb10u1).
0 upgraded, 0 newly installed, 0 to remove and 22 not upgraded.

What this command and output are telling me is: you are already on the newest version of apache2 that works with your version of Debian (the Linux distro running on the Pi). In other words, I have a limited number of choices to get this setup working:

1. Downgrade django
2. Compile libapache2-mod-wsgi-py3 from source, linking against a new enough libpython
3. Try to upgrade Debian, in the hopes that a newer version will have a newer python and newer apache2 to match it.

I rate these as follows:

1. Gross / disappointing
2. Really hard
3. My best hope.

Given those options, I think I’ll take a swing at “my best hope” first, but let’s just try to see (1) what the latest debian is and (2) what the system python3 is on that. I’ve followed this guide and succeeded in upgrading my Pi to a newer OS distro once before, but it was kind of a pain. I’d really like the answers to two questions first:

1. What’s the oldest python that works with a maintained version of django?
2. What’s the python3 that ships with debian 11 / “bullseye”, the newer version of the Linux distro running on the Pi.

For number 1, it looks like django 4.2 supports python 3.8 and 3.9 For number 2, it looks like 3.9 will be included in the upgrade. So I guess it’s time to try upgrading the Raspberry Pi again. My other concern is that the Pi is like 4 or 5 years old, and might just not work on the newer OS image. I need to know 2 more things:

1. Will my pi handle the new OS?
2. Will my django app run on python3.9?

The second one is probably easier to answer, and a no on either stops this project, so let’s answer that one on my laptop, where the django app at least works:

1. Install Python 3.9 using asdf
2. Edit my pyrpoject.toml to permit python version 3.9
3. Run poetry env use 3.9
4. Run poetry run python manage.py runserver and make sure it starts.

So far so good! Now let’s see about upgrading the whole distro on the Pi!

It took me a while to get upgraded. I had to run sudo apt install gcc-8-base to get a new enough libc that I could upgrade other things, and I skipped the command sudo rpi-update because it’s own output says, “don’t do this as part of normal upgrades,” and I sure hope what I’m doing is a normal upgrade.

I also had some issues where downloads would hang, and an apt-get install or sudo apt dist-upgrade would print the same GET line over and over again without making any progress. So I learned that if you send CTRL+C to a running apt install, it might stop and then be able to restart where it left off. I didn’t want to learn this, and I don’t recommend it, but it did happen and get me unstuck! I thought I would have to switch mirrors, but luckily just retrying everything worked.

So now, after running a full upgrade of the OS, I refresh my browser, hoping I’ll see a django page!

Instead, I see the most helpful stacktrace I have seen in my life:

File "/home/pi/pidjango/library_django/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
File "/home/pi/pidjango/djenv/lib/python3.9/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
File "/home/pi/pidjango/djenv/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
File "/home/pi/pidjango/djenv/lib/python3.9/site-packages/django/apps/registry.py", line 83, in populat
    RuntimeError: populate() isn't reentrant

This error is apparently thown by django itself

In my humble opinion, this is a terrible error message, and when I have time, I want to write a post about good error messages and why they’re so hard to write. In the meantime, let’s try to get unblocked here.

One nice thing about getting a distinctive (but completely mad!) error message on a super common framework is that I can just paste it into google, yielding this SO answer, which tells me that I need to modify some Django code to remove the raise statement and get a better error message. So I edited some core django code in Vim on the server to remove that raise statement, so that I could get a better error message. This is also not something I super recommend, but is something that worked so yay.

Now I get ModuleNotFoundError: No module named 'catalog', which is actually probably a problem with my app, since my app has a module named catalog.

This answer makes me suspect my python path is wrong.

Trying to understand this error better, I did some experiments.

1. I activated the virtual env and ran python manage.py runserver. It worked.
2. I had python print out the path it’s looking for dependencies in:

>>> import sys
>>> print(sys.path)
['', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/home/pi/pidjango/djenv/lib/python3.9/site-packages']

I poked around for a little while. At this point I was pretty frustrated, because I’ve compiled python from source twice, done a full distro upgrade, and I am still getting generic 500 pages from the webserver. After taking a break to calm down, I realized that it was a stilly error:

I had the python path wrong in /etc/apache2/sites-enabled/000-default.conf. The file 000-default.conf define the VirtualHosts that apache2 can route traffic to. Here’s the offending snippet and the fix:

<VirtualHost *:80>
    # snip ...
    # THIS PATH IS WRONG:
    WSGIDaemonProcess django python-path=/home/pi/pidjango/library_django/:/home/pi/pidjango/djenv/lib/python3.9/site-packages/ python-home=/home/pi/pidjango/djenv
    WSGIProcessGroup django
    WSGIScriptAlias / /home/pi/pidjango/wsgi.py
</VirtualHost>

I needed to change it to:

<VirtualHost *:80>
    # snip ...
    WSGIDaemonProcess django python-path=/home/pi/pidjango/:/home/pi/pidjango/djenv/lib/python3.9/site-packages/ python-home=/home/pi/pidjango/djenv
    WSGIProcessGroup django
    WSGIScriptAlias / /home/pi/pidjango/wsgi.py
</VirtualHost>

In other words, the python path was one directory too specific, and pointed into one of my apps, instead of into my whole project. With that fixed, I can at least see django-rendered pages in the browser.

I still have no static assets and no database, but those will have to wait until next time.

Till then, happy learning!
– Will

Discussion

Love this post? Hate it? Is someone wrong on the internet? Is it me? Please feel free to @me on mastodon

I used to host comments on the blog, but have recently stopped. Previously posted comments will still appear, but for new ones please use the mastodon link above.

Join the conversation!