this post was submitted on 28 Jun 2023
480 points (98.4% liked)

Lemmy.World Announcements

29079 readers
283 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn't work.

I'll create an issue for the devs and retry once it's fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..

top 50 comments
sorted by: hot top controversial new old
[–] MrPear@lemmy.world 103 points 1 year ago (1 children)

Thank you for the transparancy!

[–] GatoB@lemmy.world 15 points 1 year ago

I love post like this, another reason of why this instance is so good

[–] morelikepinniped@lemmy.world 42 points 1 year ago

Thanks for all your hard work @ruud@lemmy.world. It's not easy!

[–] OsrsNeedsF2P@lemmy.ml 22 points 1 year ago (2 children)

Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!

Dessalines:

Nutomic:

[–] imaqtpie@sh.itjust.works 6 points 1 year ago (2 children)

Yeah they've been hard at work all month. But it's also okay if things aren't ready in time. Most of the people who matter are already here.

Maybe we will blow up soon, maybe later, but the quality of content here is sufficient to drive growth regardless of whether or not we get the prophesized huge migrations from reddit

load more comments (2 replies)
load more comments (1 replies)
[–] ulu_mulu@lemmy.world 21 points 1 year ago (1 children)

is that users can’t login anymore

Ouch, that's bad :(

Thank you for your work, as always :)

[–] LazaroFilm@lemmy.world 5 points 1 year ago

I mean that’s a Tuesday on Reddit lmao.

[–] rrobin@lemmy.world 18 points 1 year ago (1 children)

As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.

We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.

load more comments (1 replies)
[–] Machefi@lemmy.world 16 points 1 year ago (8 children)

Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?

[–] ruud@lemmy.world 14 points 1 year ago

Because of a few things, really anoying bugs and the Jerboa app not working properly with older versions

[–] axzxc1236@lemmy.world 14 points 1 year ago* (last edited 1 year ago) (32 children)

Reasons I can think of:

  1. The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.

  2. Sorting is broken pre 0.18, new posts keeps flowing in.

  3. Performance improvement by removing web socket from lemmy. (which fixes 2, which is why 1. happens)

load more comments (32 replies)
[–] dragontamer@lemmy.world 12 points 1 year ago* (last edited 1 year ago) (2 children)

Federation is completely borked with .18 servers. Its very difficult for us to interact with https://lemmy.ca in any way: subscribing, upvoting, comments, posting... its all bugged.

Its maybe not that big a deal because Lemmy.world "has the most users", so in some regards its https://lemmy.ca's loss but... we need to restore reliable federation... especially before the July 1st rush IMO.

The .17 to .18 upgrade is basically a soft-defederation event, because of whatever this bug is between the two versions.

load more comments (2 replies)
[–] twistedtxb@lemmy.world 7 points 1 year ago

I suspect there will be a large influx of new users in two days, and that having Jerboa not working on .world might cause a few issues

[–] antik@lemmy.world 6 points 1 year ago

For example the frontpage is no longer being constantly filled with random posts when you set the filter to "active" or "top day".

load more comments (2 replies)
[–] Jerti@lemmy.world 14 points 1 year ago

UI: 0.18.0 BE: 0.18.1-rc.1 πŸ‘

seems to be working for me, I was already logged on

[–] itadakimasu@lemmy.world 12 points 1 year ago

Thanks for the hard work /u/ruud, we'll get there

[–] slimerancher@lemmy.world 12 points 1 year ago

Oh. Good luck with the next try!

[–] Nintendo@lemmy.world 11 points 1 year ago

appreciate the transparency. how are things looking in the back in lemmy.world (server wise)? will we get to a point where it wont require complete rollbacks on the state when a botched update gets rolled out?

[–] Sterben@lemmy.world 9 points 1 year ago* (last edited 1 year ago) (1 children)

I am glad I didn't write my last post during the upgrade process.

Maybe next time, give a warning, or maintenance notice.

Thanks for trying though.

load more comments (1 replies)
[–] naneek@lemmy.world 9 points 1 year ago* (last edited 1 year ago) (1 children)

Thanks for the transparency. Maybe it’s a good idea to have a test instance and some test cases/validation done there before updating the main instance. This is a regular process in any software/tech company/stack.

Testing should never be done directly on the prod instance.

[–] 65gmexl3@lemmy.world 8 points 1 year ago (1 children)

i think they tested it which worked fine, but it is totally different when there are users online

[–] antik@lemmy.world 6 points 1 year ago

Correct there was a test instance and that worked fine but once the live instance was upgraded everything became very slow. Can't really emulate that kind of load

[–] Guy_Fieris_Hair@lemmy.world 8 points 1 year ago (4 children)

Suddenly my Jerboa won't change from local and I can't change the sort. Not sure if this is related, but it was working fine yesterday after I updated the app. But no longer this morning.

load more comments (4 replies)
[–] Buffalox@lemmy.world 7 points 1 year ago

Linux/Firefox After upgrade many things didn't work, but they continued not to work after rollback.

Turns out I needed to clear the cache, then everything worked fine again.

[–] snargledorf@lemmy.world 7 points 1 year ago (1 children)

@ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?

With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.

[–] ruud@lemmy.world 6 points 1 year ago (5 children)

Yes, considering that. But we'll need people to use that when we will do testing...

load more comments (5 replies)
[–] Spitz@lemmy.world 7 points 1 year ago (5 children)

Jerboa is almost unusable for me at the moment. A lot of crashes, pages not loading, messages not sending etc. But I know that's inevitable at times. Restarting my phone seemed to make it worse. But I know you guys are on it! πŸ‘

[–] kratoz29@lemmy.world 4 points 1 year ago (10 children)

I recommend you this web app

https://wefwef.app

It is almost as functional as Jerboa and very impressive for a web app.

load more comments (10 replies)
load more comments (4 replies)
[–] necropola@lemmy.world 6 points 1 year ago* (last edited 1 year ago) (6 children)

Eveyrthing (UI) is extremely slow now. Not an improvement at all.

load more comments (6 replies)
[–] ekZepp@lemmy.world 6 points 1 year ago (1 children)
load more comments (1 replies)
[–] DRx@lemmy.world 6 points 1 year ago

Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.

Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though

any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?

[–] pampoon@lemmy.world 5 points 1 year ago (1 children)

I’ve found from hopping around some other instances that have upgraded to 0.18 that it is still pretty buggy. It does seem to be giving more information about the errors, instead of just failing like in 0.17, but spend any time browsing on those instances and you’re bound to be inundated with JSON and query errors. It also seems to get worse the longer you browse.

The UI changes are nice, and I do appreciate not having my feed auto-updating constantly, but I think you’d be making the right choice to hold off on upgrading until they can iron 0.18.1 out all the way. I’m not super knowledgeable about TS and Rust, but as a user it seems that switching from WebSocket created/shined a light on Lemmy’s issues with caching in general.

load more comments (1 replies)
[–] SeaJaye@lemmy.world 5 points 1 year ago

I'm so new, I'm a baby, I have no idea what I'm doing. I did just create my account, I tried using the app but it wouldn't let me. Saying server version too low. I'm sorry for my ignorance, but is this what that means? And this means we can't use the app yet right?

Also, thank you for all of this. I will be here for it all, I'm staying forever! I already love this so much more than anything else πŸ₯°

[–] TeaHands@lemmy.world 5 points 1 year ago (4 children)

Was worth a try! But yeah that was pretty bad 😬

load more comments (4 replies)
[–] Asudox@lemmy.world 4 points 1 year ago (2 children)

I am glad y'all rolled back. I had just enabled 2FA and could not login to my account. Rolling back removed 2FA so I was able to log back in. Thanks.

load more comments (2 replies)
[–] Anarch157a@lemmy.world 4 points 1 year ago

Seems 0.18 is up... and...

OH NO ! The interface changed ! Bring it back ! Bring it back... /s

Just kidding... It's looking good, so it's time to explore a little and find all the new stuff.

Nice work @Rudd, thanks for all the hard work.

[–] Shadywack@lemmy.world 4 points 1 year ago

Running one of the largest and most vibrant Lemmy instances certainly makes this deployment a great test case for the devs and the system overall. @ruud@lemmy.world this community really appreciates you!

[–] rr7@lemmy.world 4 points 1 year ago

My current session works now with 0.18.0 + 0.18.1.rc1

[–] oldLady80@lemmy.world 4 points 1 year ago

Thank you for your work!

[–] NABDad@lemmy.world 4 points 1 year ago

Thanks for the info. You're doing a great job!

load more comments
view more: next β€Ί