Will Murphy's personal home page

apt-get update doesn't work in the morning

I was trying to build a container image recently, and got some unexpected errors. My first, instinctive summary of the error was “apt-get update doesn’t work in the morning.”

The error:

root@ee0cc7ab3f4e:/# apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://deb.debian.org/debian-security buster/updates InRelease [34.8 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [56.6 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 Packages [7909 kB]
Reading package lists... Done      
E: Release file for http://deb.debian.org/debian-security/dists/buster/updates/InRelease
is not valid yet (invalid for another 3h 48min 16s). Updates for this repository will not be applied.
E: Release file for http://deb.debian.org/debian/dists/buster-updates/InRelease
is not valid yet (invalid for another 17h 12min 0s). Updates for this repository will not be applied.

occurs when building:

FROM public.ecr.aws/docker/library/rust:slim-buster

RUN apt-get update && \
    apt-get install -y libyaz-dev && \
    apt-get install llvm-dev libclang-dev clang && \
    apt-get install -f

RUN rustup target add x86_64-unknown-linux-musl

COPY . /app


RUN cargo build --release --target x86_64-unknown-linux-musl

Once, when I was working as a high school teacher, we re-connected a desktop computer that had been unplugged for a long time, and tried to get it working again. Whatever battery keeps the clock ticking when a desktop is powered off had died, and when the computer came up, it thought the current time was some date in the 80s. That computer had weird certificate errors trying to visit any website. The error above, where it says E: Release file for ... is not valid yet (invalid for another 3h 48min 16s) reminded me of the old computer lab error, where the computer was wrong about the time, and so thought all the websites it visited were presenting certs that weren’t valid yet, so I decided to ask what time it was in the container.

# in container:
$ date
Fri Nov 25 15:07:45 UTC 2022
# on host:
$ date
Sat Nov 26 06:37:27 EST 2022

After some digging, I decied it might not be the particular container, and did this:

$ podman run -ti alpine sh
$ date # container
Fri Nov 25 19:11:15 UTC 2022
$ date # host
Sun Nov 27 15:01:01 EST 2022

It looks like time doens’t advance in any container while my laptop is asleep. This seems to be a issue with podman: https://github.com/containers/podman/issues/11541. Fortunately, that GitHub thread has the answer:

$ podman machine ssh sudo date --set $(date +'%Y-%m-%dT%H:%M:%S')
Sun Nov 27 15:02:16 EST 2022
$ podman run -ti alpine date                                     
Sun Nov 27 20:02:22 UTC 2022

Now apt-get update works in the morning!

To sum up:

  1. I would sometimes work on a project with Podman, with the pattern that I’d start the Podman VM in the evening, and then come back sometime later, often the next morning, and see that apt-get update wouldn’t work in my container.
  2. After some digging, I learned that the Podman VM’s clock doesn’t run when my laptop is asleep, causing errors where it thinks that some verification process isn’t valid yet.
  3. There are a couple workarounds, such as restarting the VM or SSHing into it to update its clock.

This was a fun example of how the issue described by a user, “apt-get update doesn’t work in the morning” can be quite a bit different from the statement we’d make knowing the cause, namely, “Podman VM clock doesn’t advance while laptop is asleep.”

That’s all for today! Till next time, happy learning!
– Will


Note: recently submitted comments may not be visible yet; the approval process is manual. Please be patient, and check back soon!

Join the conversation!