apt-get update doesn't work in the morning
I was trying to build a container image recently, and got some unexpected
errors. My first, instinctive summary of the error was “apt-get update doesn’t
work in the morning.”
The error:
root@ee0cc7ab3f4e:/# apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://deb.debian.org/debian-security buster/updates InRelease [34.8 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [56.6 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 Packages [7909 kB]
Reading package lists... Done
E: Release file for http://deb.debian.org/debian-security/dists/buster/updates/InRelease
is not valid yet (invalid for another 3h 48min 16s). Updates for this repository will not be applied.
E: Release file for http://deb.debian.org/debian/dists/buster-updates/InRelease
is not valid yet (invalid for another 17h 12min 0s). Updates for this repository will not be applied.
occurs when building:
FROM public.ecr.aws/docker/library/rust:slim-buster
RUN apt-get update && \
apt-get install -y libyaz-dev && \
apt-get install llvm-dev libclang-dev clang && \
apt-get install -f
RUN rustup target add x86_64-unknown-linux-musl
COPY . /app
WORKDIR /app
RUN cargo build --release --target x86_64-unknown-linux-musl
Once, when I was working as a high school teacher, we re-connected a desktop
computer that had been unplugged for a long time, and tried to get it working
again. Whatever battery keeps the clock ticking when a desktop is powered off
had died, and when the computer came up, it thought the current time was some
date in the 80s. That computer had weird certificate errors trying to visit any
website. The error above, where it says E: Release file for ... is not valid yet (invalid for another 3h 48min 16s) reminded me of the old computer lab
error, where the computer was wrong about the time, and so thought all the
websites it visited were presenting certs that weren’t valid yet, so I decided
to ask what time it was in the container.
# in container:
$ date
Fri Nov 25 15:07:45 UTC 2022
# on host:
$ date
Sat Nov 26 06:37:27 EST 2022
After some digging, I decied it might not be the particular container, and did this:
$ podman run -ti alpine sh
$ date # container
Fri Nov 25 19:11:15 UTC 2022
$ date # host
Sun Nov 27 15:01:01 EST 2022
It looks like time doens’t advance in any container while my laptop is asleep. This seems to be a issue with podman: https://github.com/containers/podman/issues/11541. Fortunately, that GitHub thread has the answer:
$ podman machine ssh sudo date --set $(date +'%Y-%m-%dT%H:%M:%S')
Sun Nov 27 15:02:16 EST 2022
$ podman run -ti alpine date
Sun Nov 27 20:02:22 UTC 2022
Now apt-get update works in the morning!
To sum up:
- I would sometimes work on a project with Podman, with the pattern that I’d start the Podman VM in the evening, and then come back sometime later, often the next morning, and see that
apt-get updatewouldn’t work in my container. - After some digging, I learned that the Podman VM’s clock doesn’t run when my laptop is asleep, causing errors where it thinks that some verification process isn’t valid yet.
- There are a couple workarounds, such as restarting the VM or SSHing into it to update its clock.
This was a fun example of how the issue described by a user, “apt-get update doesn’t work in the morning” can be quite a bit different from the statement we’d make knowing the cause, namely, “Podman VM clock doesn’t advance while laptop is asleep.”
That’s all for today! Till next time, happy learning!
– Will
Love this post? Hate it? Is someone wrong on the internet? Is it me? Please feel free to @me on mastodon
I used to host comments on the blog, but have recently stopped. Previously posted comments will still appear, but for new ones please use the mastodon link above.
Discussion