this post was submitted on 20 Jun 2023
76 points (100.0% liked)

..:: tchncs ::..

1280 readers
1 users here now

Your friendly https://tchncs.de community! Discuss whats happening in the tchncs world and/or just use it as a community forum.

German and english allowed.

If you are looking for a way to support tchncs, please check out https://tchncs.de/donate


founded 1 year ago
MODERATORS
 
Hello there friendly users!

Update 3:50pm CEST
The container has been moved, most work on the new host has been completed. There are some permanent data errors on the original storage pool, but it does not like it affected this linux container. Please give DNS and federation a moment.


Some (but only a few) of you have experienced an outage today around 5-8am CEST. This – as far as my current findings go – happened due to a filesystem panic. So far, i only experienced such a thing on virtual servers with a certain amount of load.

Even tho relatively speaking, i didn't feel like there was much to worry about with the current setup and its utilization, i feel like when looking at a potential second wave from Reddit, this instance should be prepared – so i just did a thing:

What we are moving to

We will be moving from a Netcup RS 4000 G9.5 KVM with 32 GB memory and 10 dedicated EPYC cores, (that was meant for various smaller tchncs services such as XMPP, Email, Clouds, Websites and stuff and lots of headroom), to a Hetzner serverauction baremetal thing with an Intel Core i7-7700, 2x SSD M.2 NVMe 512 GB, 64 GB DDR4 Mem, 1 Gbit Networking in Finland.

I plan to migrate today as soon as the thing is ready and configured. There will be a downtime of up to 1 hour.

Lemmy will share its seat on the machine with the tiny Calckey instance that was deployed around the same time.

Increased monthly cost: around 40 EUR


Note on the thing Meta is developing with ActivityPub federation support:
I am not a friend of Meta and am happy to keep it out of my life an off of my servers. However there are lots of rumors and barely any confirmed informations around the project. I will act accordingly once more is known. I also won't make any exceptions in terms of enforcing our rules. However i won't participate in signature lists for pre-blocking everything Meta and run around in circles, screaming in panic.

top 24 comments
sorted by: hot top controversial new old
[–] WenAmon@discuss.tchncs.de 14 points 1 year ago

Thank you for your time and effort.

[–] ruapho@discuss.tchncs.de 12 points 1 year ago (1 children)

Thanks for investing your time and money. Btw: the Donate page is down (502 Error)

[–] milan@discuss.tchncs.de 9 points 1 year ago

sorry about that, the website is back up – and thanks for considering :)

[–] thomholwerda@kbin.social 6 points 1 year ago

Excellent position on the Meta thing, Milan! It's the only sane thing to do at the moment, I think.

Thanks for all the hard work!

[–] BeliefPropagator@discuss.tchncs.de 4 points 1 year ago* (last edited 1 year ago) (1 children)

Is there a metrics/grafana page where I can see the bandwidth and compute that the server uses? Do you have insights into that? Where do you see the bottlenecks in the current system?

[–] milan@discuss.tchncs.de 3 points 1 year ago (1 children)

Not right now, sorry, but i might add something like this (again) in the near future. There was some memory leaky behavior and a crying ZFS of this kind never happened to me on my dedicated hosts so far. Otherwise the previous system was fine... hard to predict the next incoming wave and stuff tho.

Np. Don't worry about it, I don't want to hassle you into doing extra (potentially) unnecessary work. Interesting that ZFS was leaking - I have no experience with it, I just heard that ZFS in general requires a lot of RAM to be a reasonable filesystem choice. Are you sure it was a leak, or could it have been e.g. caches filling up as well?

[–] squirrel@discuss.tchncs.de 4 points 1 year ago

Wohoo, we are back

[–] ivanhoe@discuss.tchncs.de 2 points 1 year ago* (last edited 1 year ago) (1 children)

Was there any data loss caused by the outage? I can't find a post anymore that I made a while ago. It's nothing important, I'm just curious.

[–] milan@discuss.tchncs.de 3 points 1 year ago

there shouldn't be data loss from before the incident...

[–] heimchen@discuss.tchncs.de 2 points 1 year ago (1 children)

How much does the server cost all in all monthly (or are the 40€ after the increase and not just the difference between previous and after)

[–] milan@discuss.tchncs.de 4 points 1 year ago* (last edited 1 year ago)

The server that was just rented is around 38 € per month. I am not sure if this was including tax or not... (maybe with a setup fee tho). There are also dynamic costs for media storage, but there are no per-container statistics of this kind... it shouldn't be much more than an euro or something right now, i don't know.

..the old server will remain in the tchncs infra.

[–] elektronaut@discuss.tchncs.de 1 points 1 year ago (1 children)

Seems like I cannot post notes with attached images on Calckey. Text-only posts do work though. @elektronaut@procial.tchncs.de

[–] milan@discuss.tchncs.de 1 points 1 year ago (1 children)

sorry about that. it should now be fixed

[–] elektronaut@discuss.tchncs.de 0 points 1 year ago (1 children)

Never mind. Posting notes with attachments works again now. 👍🏻

[–] milan@discuss.tchncs.de 2 points 1 year ago

No never mind needed: missed a little thing in the domain config. Glad that you can confirm it being working now :)

[–] cracks@kbin.social 1 points 1 year ago

Didnt experience outage but there are some Errors time to time.

load more comments
view more: next ›