I'm an admin on The Common Room and things have been fine until today, when I noticed that we're no longer receiving streams from the broader fediverse (on the federated timeline).
I deleted all the previous relays and added new ones/re-entered the old ones, and nothing is working still. The relays are all stuck on 'waiting for relay's approval'.
I'm not sure how to begin debugging this issue. Sidekiq is throwing 504s and 401s (suggesting some relays are down, which is reasonable, but it's the 401s that really bug me).
Has anyone experienced this before? How would you recommend I debug it?
Screenshots below:
Above: List of relays added. You'll see they're all stuck on 'waiting'.
Snapshot of the Sidekiq retries queue, in case it may offer some insight.
Thank you!
- I've run
systemctl restart {nginx, mastodon-web}to no avail. - I've killed pending queues and retried new ones.
- I've run
journalctl sidekiq.service -fto get a live tail of errors, but nothing interesting.
Worth noting: this error may/may not have arisen after we migrated the front-end to use Soapbox's frontend.
Propably most of those relays are dead or not accepting new connections because of high load... but I can confirm that I got one server which is pending and we cannot solve issue with relay admin - it stuck. We disabled enabled several times nothing. I tried to journalctl to find issue nothing. Sidekiq nothing. So on the other hand I can confirm that there is issue with RELAYS on mastodon v 4.0.2