Repliqate only polls repliqate.enabled labels for use on startup #23
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
(I'm just staring to test repliqate today.)
repliqate is only checking for repliqate.enabled: true docker labels on initial startup. It is not continuously polling for updates.
Today I added repliqate docker labels to one of my containers for testing. I started up repliqate and it correctly showed me in the logs that it was supposed to backup the container at xx:xx time. I then went and added repliqate docker labels to a few more containers and re-installed them. I watched the reqpliqate logs and four hours later Repliqate.Services.ScheduleManager did not know to backup my other apps. Upon restarting repliqate, the labels were discovered and the schedule set to backup the containres.
IMHO: This is a serious bug.
Interesting issue. I'll take a look at this. It should be fully active and listening on changes to Docker itself through the callback system of Docker's sock.
Would you mind posting your repliqate logs here please?
Sure thing.
Weird, I can see repliqate is connecting to the Docker sock. Could you post your repliqate docker compose if you could? I'd like to see how you've bound the docker sock
Sure thing.
(Can be used as another example of same request in https://github.com/lminlone/repliqate/issues/21#issuecomment-3632250588)
vars:
task:
I'm going to go add repliqate labels to some more of my containers and re-deploy them; then I'll post some more logs.
Two setup for repliqate currently:
I just added the repliqate true label for two more containers, acmenator and gotify.
I
docker stop'd anddocker rm'd both acmenator and gotify.I ran my Ansible script to redeploy both acmenator and gotify with the newly added
repliqate.enabled: truelabels.I just looked at the repliqate logs in dozzle and from docker logs and nothing has updated. It's been roughly five minutes or so.
ENTIRETY of the logs:
It's 3+ hours later and the logs remain the same.
(I won't check any more.)
Thanks for that info; repliqate should pick up any changes almost immediately after you've started the stack. I don't see any issues with it connecting with the docker sock, so it's difficult to determine what the issue could be.
What version of Docker are you using and are you using it directly to make updates or is Ansible calling into it to make those stack/container changes?
The Ansible script is the Ansible community.docker.docker_container module. I searched for a better description of what happens then what I could write and found: "When this playbook is executed, Ansible connects to the docker_hosts, uses the community.docker.docker_container module to interact with the Docker daemon on those hosts, and creates or manages the my_nginx_container according to the defined specifications. If the container doesn't exist, it will be created. If it exists but its configuration differs from the playbook, Ansible will update it to match the desired state."
I'm using Docker version 29.1.2, build 890dcca.
OK. Now for the fun part that I'm hoping makes you go "Hmmmmmmmmm..."
About 24 hours ago I added repliqate enabling labels to my acmenator and gotify containers and recreated them as previously posted. By design, repliqate should have read the new labels from the docker socket and prepped to do backups. According to the logs, it didn't notice the change. (No new info here, all rehash). Repliqate first became (via announcement in the logs) of the containers with repliqate enabled labels at 3:01 AM after the VM was shut down at 3:00 AM and for the scheduled Wed morning backup. This supports my theory that repliqate is only (successfully) polling the docker socket for labels when the repliqate container starts.
Here's the full follow-up logs:
THOUGHTS on Docker versions...
There were some API changes with Docker v29. I've had some other programs have compatibility problems. (If I remember correctly and pressed for time so can't verify ATM) I think current API is v1.55 and minimum required compatibility changed to v1.44. I had some other apps with issues complaining about API v1.24/5 after upgrading to Docker v29 a few weeks ago.
Thanks for all this info, it's really handy! I'll see if I can upgrade my test cases for the new Docker. I suspect that this is the issue as I'm on Docker v27 with API v1.46.
I'll put a bunch more logging around that area too as a precaution for the next patch update.
I have been testing locally (Windows) with Docker 29.1.2 | API 1.52 and it seems to be functioning correctly.
Once you have a chance, I would greatly appreciate you trying v0.2.3 (doesn't contain any fixes, only extra logging) with the environment variable
LOG_LEVELset toDebugplease? That would help hopefully diagnose the lack of on-demand updates.But from your last logs it looks like it cannot even connect to the Docker sock? Could you confirm you have it bound properly?
Fixed now in v0.2.4.
Appreciate your time @bcurran3, it's been incredibly helpful.
Glad to help out.
I'm on v0.2.4 thanks to Watchtower at 5 AM. (side note: https://github.com/lminlone/repliqate/issues/25)
I just added repliqate enabled label to portainer and rebuilt it.
All lines in the repliqate log are from 5 AM when it restarted, no updates since.
I did not change log_level (request for v0.2.3 and assumed rescinded for v0.2.4 expecting the problem to be fixed).
-> I do have a theory that the problem MAY be in my environment (all this time?). <-
Busy today, will try testing on another server, most likely tomororw, and report back the results.
There was an issue that I overlooked, fixed in 0.2.4 with the filters for events not operating as expected. I figured that would have been your issue...the v0.2.3 log level still exists (and will continue to exist) for v0.2.4, so you should be able to turn on debug logging when you have time, for me to dissect at least.
What in particular do you think might be an issue with your environment?
I've been busy working on my unpublished project (which incorporates/integrates repliqate). When I have a chance (hopefully later this week) I will test and report back.
I haven't had a chance yet to go back and test with repliqate (Mainly because I can't really make use of it until bind mounts are supported without refactoring 280+ Ansible roles.) but the issue might be related to ghost containers. Ghost containers were causing me issues with Homepage and Portainer-CE. I'm guessing the problem was creeping in on repliqate as well. You can read my experience about it here: https://github.com/portainer/portainer/issues/12959