datahoarder

6777 readers

1 users here now

Who are we?

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

We are one. We are legion. And we're trying really hard not to forget.

-- 5-4-3-2-1-bang from this thread

founded 4 years ago

MODERATORS

archivist@lemmy.ml

How do you handle backup? (programming.dev)

submitted 3 months ago by anzo@programming.dev to c/datahoarder@lemmy.ml

12 comments fedilink hide all child comments

cross-posted from: https://lemmy.dbzer0.com/post/26278528

I'm running my media server with a 36tb raid5 array with 3 disks, so I do have some resilience to drives failing. But currently can only afford to loose a single drive at a time, which got me thinking about backups. Normally I'd just do a backup to my NAS, but that quickly gets ridiculous for me with the size of my library, which is significantly larger than my NAS storage of only a few tb. And buying cloud storage is much too expensive for my liking with these amounts of storage.

Do you backup only the most valuable parts of your library?

top 12 comments

sorted by: hot top controversial new old

[–] anzo@programming.dev 4 points 3 months ago

I'm crossposting because some comments were interesting...

[–] thejoker954@lemmy.world 3 points 3 months ago (1 children)

I live dangerously. Ive no backups.

[–] BearOfaTime@lemm.ee 1 points 3 months ago

I tip my fedora at you sir

[–] thirteene@lemmy.world 1 points 3 months ago* (last edited 3 months ago)

4 bay nas 22tb drives = 88tb as a single volume. Backed up to offline single storage drives that are timestamped. Anything newer than that timestamp gets rsynced/robocopied to a new drive until it's full., update timestamp, swap drive and rinse repeat. If a drive fails, just replace it and copy the contents of the backup drives to master and restored. Alternatively you can set a single disk tolerance raid.

[–] Nogami@lemmy.world 1 points 3 months ago* (last edited 3 months ago)

Unraid checking in. 150TB ish. Dual parity for media files on XFS drives. Can lose 2 drives before array data is at risk, and even then if a drive fails, I’d only lose the data on that drive not the entire array.

Important data is on ab encrypted ZFS array which is snapshotted and replicated hourly to a separate backup server and replicated one more time weekly to 3.5” archive drives which are swapped out to a safe deposit box every few months.

I used to use rsync to do it but snapshot replication with sanoid/syncoid does the same thing in a tiny fraction of the time by just sending snapshot deltas rather than having to compare each file.

[–] d_ohlin@lemmy.world 1 points 3 months ago* (last edited 3 months ago)

Used to use rclone encrypted to gsuite, until that gravy train ran out lol.

Now I've got 3 Unraid boxes (2 on site, 3rd off) and a metric shitton of drives I've accumulated over the years.

[–] leviticoh@poliverso.org 1 points 3 months ago (1 children)

@anzo

an unholy ensemble of 4 external drives and 2 cloud storage providers managed with git annex
capacity: 3 TB + "pay as you go"
available: 1TB
used: 1.01TB
the drives were originally 5, then 1 failed and added the "pay as you go" s3 provider to pick up the slack
git annex handles redundancy and versioning, i configured it to keep 2 copies of every file across the 4 drives and the s3 provider, then i got 1tb of free onedrive space from my university and i'm in the process of making a full backup of everything in there
not really backup as much as redundant storage, but i think it's close enough

if anyone wants to roast my setup please do

[–] christophski 1 points 1 month ago (1 children)

What cloud storage providers would you recommend?

[–] leviticoh@poliverso.org 2 points 1 month ago (1 children)

@christophski
i prefer not to recommend specific storage providers, as i can't ensure they'll keep being good in the future and won't suddenly fail taking all data with them.

Still, like other data hoarders probably already told you, ensure that a single provider doesn't hold the only copy of your data and that the files you put on them are encrypted, so that weirdos don't go snooping in your privates.

I'd also advise choosing providers that support standard protocols like s3, sftp, webdav or similar, so that you can use an external tool to manage your data and migrate more easily in case you need to switch.

now, it always depends exactly what you need it for, i was talking about keeping personal files i care about in the cloud, you could need something different for another purpose, but it would get too long for a comment on lemmy

[–] christophski 1 points 1 month ago

Thanks!

[–] MangoPenguin@lemmy.blahaj.zone 1 points 1 month ago* (last edited 1 month ago)

I only backup important stuff that I can't replace, so photos and videos I've taken, documents, etc.. So far about 1.5TB of space used by that.

Media and other stuff I downloaded online doesn't get backed up, it's not important and can be replaced.

[–] some_guy@lemmy.sdf.org 1 points 3 months ago* (last edited 3 months ago)

Synology NAS

Edit: plus two SSD backups. I need to re-engage Backblaze for offsite.