important instance shit

I defederated us from two lemmy instances:

• exploding-heads: transphobia
• basedcount: finally I get to ban most of r/PoliticalCompassMemes in one go

we suffered some extremely unexpected downtime while I deployed a trivial change (a reverse proxy from http://awful.systems/archives to http://these.awful.systems/archives) to prod

the downtime was unrelated to the deployment change; instead, it seems like lemmy-ui started crashing because it couldn't render the app icons it uses when saved as a home screen app on mobile. it uses a fairly heavy dependency to do this, and has no error handling in case the source icon data is corrupt, which causes it to crash on every request (resulting in a 503 Service Unavailable error for everyone who tried to access awful.systems during this outage)

since I don't know how that corruption occurred or why it was persistent (the app icon data should be fully static as part of the Nix store as far as I know), so until I can dig in I've disabled generating app icons for our instance. since it seems like we're the first ones to hit this bug, I'll do my best to keep the patch upstreamable so other lemmy instances can benefit from the fix

see this lemmy-ansible github issue for the fix; basically, our web server now knows how to handle activitypub traffic in a more conforming way

to interact with us from mastodon:

• find the community you want to subscribe to here. note its real name -- that's the name in the sidebar after the !
• search for @communityname@awful.systems in mastodon
• follow that user and enjoy our posts over there! replying and boosting should work ok, no guarantees for anything else

as for interacting with mastodon from here, I think you can paste mastodon URLs into our search and it'll maybe work? someone try that

big update, awful.systems is now a federated lemmy instance. let me know if anything looks broken! here's what to expect:

• to pull up an awful.systems community on another instance, just paste that community's URL into the other instance's search bar
• federation with other lemmy instances should work, and probably kbin too? there's no way I can find to pull in pre-federation posts on remote instances though, so send your friends here to read the backlogs
• we can't federate with most of mastodon right now because lemmy doesn't implement authorized_fetch, which is a best practice setting for mastodon instances. if your instance doesn't use it, try entering something like @sneerclub@awful.systems into your mastodon search; lemmy communities are represented to mastodon as users
• this is pretty much an experimental thing so if we have to turn it off, I'll send out another post
• reply to this post with ideas for moderation tools and instances you'd like to see blocked (and a reason why) and we'll take action on anything that sounds like a good idea

federation was made possible by

• lemmy's devs skipping their release process and not telling anyone 0.18.2 was released on friday? so we're on 0.18.2 now
• updating all of the deployment cluster's flake inputs just in case
• @dgerard@awful.systems shouting yolo

I was gonna do this quietly since I was doing it mostly for security fixes, but now I guess I gotta announce that I deployed lemmy 0.18.1 to the awful.systems cluster. changes include

• sweet christ did this UI get smaller and uglier? whose idea was this.
• we have more theme options! most of them are terrible. there is a vaporwave theme I kinda like in a geocities way. if you come here and it looks like geocities I switched to that one
• they fixed like 3 out of the 4 webdev 101 security holes they left in the code
• there's some small new UI features?
• sometimes they just make changes for no reason
• let me know if anything looks broken

I rolled out some minor but important updates to the deployment cluster just now:

• email notifications are now enabled in production. let me know if this tanks performance. also we’re on a fairly limited email plan so I’ll post an update if we exceed its limits (which’ll break notifications again)
• we now have a staging deployment and a development deployment, which let me make these changes without taking production down
• lemmy-ui in production is now running in production mode, which should improve performance slightly
• late update: awful.systems now reports a correct lemmy backend version, so lemmy mobile apps should work. I confirmed that mlem on ios works, but let me know if jerboa or anything else is broken

I just finished a migration that doubled the resources awful.systems has available. let me know if I fucked anything up and didn't notice

changelog for this deployment:

• more.awful.systems, a Hetzner CPX31, was added to the cluster
• all dynamic data was migrated from these.awful.systems to more.awful.systems
• the load balancer target was swapped from these to more
• now I can throw up a maintenance page for next time I need to do this

this instance runs on open infrastructure. the code that deploys awful.systems is available here.

right now I've got the following planned for the awful.systems cluster:

• ~~in addition to the current prod lemmy deployment, split off staging and dev. staging will be used to function check infrastructure updates before they hit prod. dev will be used for feature development.~~
• ~~add a maintenance mode to prod that shuts off the lemmy services and replaces every route with a maintenace page. this will be necessary for big moves like host migrations or storage expansion that'll take the database offline~~
• ~~make the backend return a damn version so the lemmy apps don't break? I'm guessing this broke because nix deletes .git when pulling sources. this can probably be fixed lazily using keepGit or properly with a patch to lemmy's version detection~~
• start work on a less janky alternative to lemmy-ui, which will be deployed to dev until it's worth using and hopefully mostly not broken (ima call it lessjank)
• also start work on better moderation tools, implemented in both lemmy-server and lemmy-ui
• ~~probably migrate prod to a bigger hetzner host -- this'll take awful.systems offline for a little bit as I restore the database into the new system~~
• ~~eventually set up sendgrid? email notifications actually working will probably be beneficial ~~

if you'd like to contribute, contact me. the deployment parts of awful.systems are written in nix, and everything else will be rust