Another day, another update.
More troubleshooting was done today. What did we do:
- Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
- @cetra3@lemmy.ml created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
- We started using this image, and saw a big drop in CPU usage and disk load.
- We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a
return 404in nginx conf for/api/v3/ws. - We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
- We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~
proxy_next_upstream timeout;~~max_fails=5in nginx.
Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.
Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!
And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!
And thank you all for your patience, we'll keep working on it!
Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.
Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.
Can we have an update on the status of Lemmy.world and how close ties we are going to have with Meta's threads? Threads is going to support ActivityPub, but time has shown that this is an attempt to try to kill this open platform and eventually replace it with theirs once they get everyone in their ecosystem. (Embrace, Extend...extinguish) Mastodon has said today that they don't mind sleeping with vipers when their demise / dissolution is in Meta's best interest.
Please tell me we are defederating from Meta....or let us know what to expect
EDIT: I originally stated that Mastodon told them to fuck off, but I got confused with Fosstodon (who did that). Mastodon doesn't mind being in bed with Meta
Where have you seen Mastodon formally state they have no interest in working with them?
I'm genuinely asking because I'm relatively new to Mastodon and Lemmy and want to be as informed as possible with this whole Meta situation. And just to be transparent, I'm not a Meta fan at all, to the extent I've never had an account with any of their products
I did read this official Mastodon blog post today...
https://blog.joinmastodon.org/2023/07/what-to-know-about-threads/
and my question to you is because I'm not seeing as quite aggressive a stance as you mentioned.
I realized I made a mistake and got Mastodon and Fosstodon confused.
You may be right and as sad as it is...Meta wanting to work with the federated network should be seen as two things... an attack on the ActivityPub standard via infiltration...and that Meta sees the Fediverse as a threat to its own closed ecosystem.
We need to be on the defense and protect our platform and standard from corporate meddling and fuckery which Meta will absolutely not hesitate to do.
Yeah, even if meta is acting in good faith, looking at what it's like here now vs Reddit last week vs Reddit a decade ago has me thinking that I don't really want to be part of a site with such a large community. It's hard to put into words exactly what it is about this place that is nicer than Reddit, maybe it's the difference between the kind of person that left and the kind of person that stayed, maybe this place just isn't popular enough to be targetted by a certain kind of shittiness.
Whatever it is, I'm worried that meta will bring a bunch of users that have that thing that made fb and Reddit less good over time.
And then there's also the angle of anyone can create an instance and suddenly gains a lot of visibility into user activity. Especially a company like meta, though any company could quietly just start sucking up that data. Maybe the fediverse needs different levels of trust between nodes instead of an all or nothing approach. Though tbh I'm still not very familiar with the implementation details or how much sense that even makes for it.