derekp7

snebu

C2013–2021

derekp7 / snebu

snebu

Simple Network Encrypting Backup Utility

Explore on GitHub Read Hacker News discussions

3Distinct people

16Mentions

0Explicit recommenders

111GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Simple Network Encrypting Backup Utility

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

andrewchambers/bupstash · linked in 3 of the same discussions
borgbackup/borg · linked in 3 of the same discussions
dpc/rdedup · linked in 3 of the same discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year16 dated mentions

Observed mentions: 2013-02-05 03:08:46 UTC to 2021-11-13 14:35:38 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
2013	2013-01-01 00:00:00 UTC	2014-01-01 00:00:00 UTC	4	0	4
2014	2014-01-01 00:00:00 UTC	2015-01-01 00:00:00 UTC	0	0	0
2015	2015-01-01 00:00:00 UTC	2016-01-01 00:00:00 UTC	3	0	3
2016	2016-01-01 00:00:00 UTC	2017-01-01 00:00:00 UTC	0	0	0
2017	2017-01-01 00:00:00 UTC	2018-01-01 00:00:00 UTC	0	0	0
2018	2018-01-01 00:00:00 UTC	2019-01-01 00:00:00 UTC	0	0	0
2019	2019-01-01 00:00:00 UTC	2020-01-01 00:00:00 UTC	2	1	1
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	1	1	0
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	6	0	6

What Hacker News says

16 sources

sandreas·Nov 13, 2021

Read the rest

ly use restic while technology wise (speed, only restore needs password) i would prefer rdedup which is an impressive piece of software but unfortunately without file iterator... :-)

In Restic – Backups Done Right

sandreas·Jul 26, 2021

Read the rest

ly use restic while technology wise (speed, only restore needs password) i would prefer rdedup which is an impressive piece of software but unfortunately without file iterator... :-)

In Deduplicating Archiver with Compression and Encryption

derekp7·Mar 19, 2021

Since I've had a handful of users ask about cloud storage for Snebu, Would you be interested in adding Snebu as a supported protocol? It should be similar to how you currently support Borg. For Snebu, the client runs find and tar, sending results via ssh to the snebu binary on the remote host. And more recently client-side public key encryption support has been added via a client-side filter called "tarcrypt". Ideally, a customer would use Snebu to back up to a local device on their network (for example a Raspberry Pi with a large USB drive attached), and then use Snebu's efficient replication to send deltas to the cloud-hosted server. Client files are stored individually (deduplicated) on the Snebu server, and metadata is in an SQLite DB (advantages over Borg is more open standards for the data storage and public-key encryption, dis

Read the rest

advantage is file-level instead of block-level deduplication and a project that isn't as widely used). If you are interested, I would be more then happy to have an extended discussion with you going over implementation options, and updating the client side script to make it work better with your service. (https://www.snebu.com, https://github.com/derekp7/snebu, and the tarcrypt extensions to tar are described at https://www.snebu.com/tarcrypt.html).

In Interview with CEO of rsync.net: “no firewalls and no routers”

derekp7·Feb 16, 2021

The code is on github.com/derekp7/snebu if you want to take a look. The next task on my list is to write up developer / contributor documentation on how the whole thing is put together along with various tricks (and simplified demonstration code). But specifically, look in "snebu-main.c" that is where the opendb function is (so you can see the pragma statements), and there is a busy_retry function that gets referenced (all it does is sleep for .1 seconds). I believe that you don't need the busy-retry function, if you use the built-in busy handler, but I'm not really sure and don't want to take a chance and break working code. For the temp tables, look in snebu-submitfiles.c -- the function at the top handles the DB operations, one towards the bottom handles the non-DB tar file consumption operations, and there is a circular queue in the

Read the rest

middle to handle buffering so the data ingestion can keep going while the data is getting flushed (these three run as separate processes). I should learn threads, as there may be more flexibility in that, but not comfortable enough with thread programming yet.

In Many small queries are efficient in SQLite

sandreas·Jan 3, 2021

Read the rest

le technology wise (speed, only restore needs password) i would prefer rdedup which is an impressive piece of software but unfortunately without file iterator... :-)

In Encrypted Backup Shootout

derekp7·Jan 3, 2021

Yes, that is correct. You can either have the backup server ssh into your clients, or set up an account on the backup server for each client to use. In that case, 'snebu' is installed owned by the user ID snebu, and is set-uid to that ID. When starting up, it checks if your UID is different than your EUID. If so, it looks up your UID in a table to determine what you can do (listbackups, restore, newbackup/submitfiles, expire, purge), affecting which backup sets. Back on the pull backups, the backup server ssh's to a client (optionally using a non-privliged user), then sudo's to root (or if backing up files owned by a specific account, can sudo to that account). And no software is required on the client in that case (except tarcrypt if doing encryption). So it is really easy to set up backups on a fleet of servers. BTW, I see in your

Read the rest

profile that you are a C++ programmer -- feel free to drop by the github repository (github.com/derekp7/snebu) if you have any issues or ideas (I have Github discussions enabled also for free form chat).

In Encrypted Backup Shootout

derekp7·Dec 18, 2020Submission

Ask HN: Help with a design review for encrypted TAR files program "tarcrypt"

I have created a program called tarcrypt which takes the output of a standard tar file (focused on Gnu tar currently), and adds compressed/encrypted extensions. This encrypts the file contents while providing standard TAR headers (with extended values in the PAX header fields). The purpose is to be able to send the encrypted tar file to backup servers which ingest a tar file input, and maintain a data catalog DB for managing the backed up files. The documentation (including file format, security discussion, and usage) is at: https://www.snebu.com/tarcrypt.html and the code is in the Snebu code repository at https://github.com/derekp7/snebu. I'd like to make sure I don't have any major design blunders prior to finalizing the release (of both tarcrypt and the updated Snebu code). All code is GPLv3.

3 points · Open discussion

derekp7·Oct 31, 2019

There is also Snebu (www.snebu.com, Github repo at github.com/derekp7/snebu). I wrote this about 8 years ago for my personal needs, because rsync / snapshot backups were running out of steam for me. What's different from Restic or Borg/Attic, is that the main "smarts" lives on the backup server (the Snebu binary), whereas the client side just uses find / tar to serialize the data. Backend side compresses, deduplicates, snapshots, handles a number of systems easily, and is fairly fast due to using SQLite for the metadata catalog. A couple weeks ago I finished a client-side encryption module, got another weekend's worth of work to integrate encryption support into the back end. The client-side encryption module is already in the Git repository, but there is no documentation for it yet. My next project is a Web based GUI (maybe an Electron b

Read the rest

ased client-side GUI too, depending on what makes sense). Oh, and the most recent release supports granular user permissions, so you can grant a host (via an SSH account on the backup server) permissions to create backups, but not delete them. Or have an administrative user that can expire old backups but not read backups, for example. Can help with thwarting crypto viruses that try to delete backups.

In Restic – Backups Done Right

derekp7·Sep 17, 2019Submission

Show HN: Snebu, a fast snapshot compressing deduplicating backup for Linux

5 points · Open discussion

derekp7·May 21, 2015

I wrote my own backup system, called Snebu (http://www.snebu.com hosted on github at http://www.github.com/derekp7/snebu). It works a lot like rsnapshot, but uses an sqlite3 DB for metadata storage and also does streaming compression (and encryption filter module is also coming soon). Client side resources are light -- just requires a system with gnu tar and gnu find, along with a bash shell for running the client-side script.

In Ask HN: What's your backup setup? Manual or Automated?

derekp7·Apr 24, 2015

Snebu ( http://www.snebu.com and https://github.com/derekp7/snebu ) is a Linux backup system that works kind of like rsync snapshot backups (i.e., similar to rsnapshot). Except it stores backup sets meta data in an sqlite database, and the files are stored as lzo compressed objects. Other than the standard features such as backup retention schedules, it has a number of small features that I haven't seen in a lot of similar backup utilities: * Path rewriting (grafting) -- this lets you mount an LVM snapshot somewhere, and the backed up copy can be stored referencing the original path (i.e., if you backup a snapshot of /srv/mydbfiles mounted on /mnt/snapshots/mydbfiles, you can tell it to store it on the backup server as if it was backed up from /srv/mydbfiles). This also works the other way for restoring files. * Safe expiring of old ba

Read the rest

ckups -- Let say you haven't backed up a particular host for a number of months (i.e., it has been decommissioned). When you tell the system to expire all backup older than 6 months, it will by default keep the 3 most recent backups [this number is tunable on the command line]. * Multiple user / granular permission support [ in the development branch, should make it in the next release ] -- You can give a given user ID the ability to backup but not restore, or backup and restore but not purge old backups, etc. That way if a host is compromised, you can ensure that an attacker can't delete backups from that compromised host. * If used in client/server mode, you can back up a client from the server without installing anything other than an SSH key (it just calls GNU find/tar on the client). Things missing that are currently being coded up: * Client side encryption * Multiple backend support (to support tape or cloud replication) * Local shadow copy -- lets you back up a laptop to a network server, but also do incremental snapshots to a local USB device when not on the network, and sync that to the network copy periodically.

In Ask HN: Brag about open-source projects you're proud of

derekp7·Apr 12, 2015

For me, the method I've found most comfortable are snapshot style whole disk backups. The common go-to solution is rsync/snapshots (ala rsnapshot or similar utilities), which I've used in the past. However this only does partial deduplication (it deduplicates files that are in the same path from one backckup to the next), and it doesn't compress. So to remedy that I wrote Snebu (http://www.snebu.com), which stores lzo-compressed files on the target using sha1 hash for the file name (somewhat like Git does), and has an sqlite DB to store the metadata (file paths, ownership, backup set name/date, etc). Therefore you get full file-level dedup even across multiple clients writing to the same backend. On the client side, it uses standard GNU find and tar, so you can use it on Windows with the help of Cygwin. Each backup set has a retenti

Read the rest

on schedule label (daily, weekly, etc) to make it easy to selectively purge old backups (i.e., keep the current 10 daily, 6 weekly, and 12 monthly backups). I've got the client side script tweaked so that you can run it as a standalone setup (backup to a local USB drive), or backup to a remote Linux server via a non-privileged account, or have the remote backup server pull backups from your clients. The features that are currently in development for the next version include client-side encryption, and pluggable backend replication, so you can replicate your backup sets to cloud providers, or tape, or to another Snebu installation. I can definitely use some more feedback, probably need to do another round of documentation improvements, but overall the code seems quite stable. The project sites is up on github (github.com/derekp7/snebu) if you are interested in contributing.

In Ask HN: What are your personal backup / sync strategies?

roryokane·Mar 31, 2013

Link: https://github.com/derekp7/snebu

In Today is World Backup Day

derekp7·Feb 26, 2013

I used to use a homebrew script that implemented rsync/snapshots (that is, rsync data to a directory on another server, then "cp -al" all files to create hard links into a directory called "server-datestamp-weekly" (or daily, monthly, etc). After running into scaling issues (esp. when needing to find a specific file that I didn't have a date or server name), I wrote (and open-sourced) Snebu. This works effectively like rsync/snapshots, but it stores the file metadata in an sqlite database (so there is very little setup complexity / maintanence), and individual files are stored compressed in a vault directory, named by the SHA1 checksum of the file. This gets cross server file-level deduplication for free. This is somewhat similar to how git stores files. Just recently posted a 1.0 release, although I need to improve the documentation a

Read the rest

bit and post a few more front-end scripts (and finish the in-app help pages). But would love to get some feedback on it. The site is www.snebu.com if you are interested (this points to the Github-hosted page, the project page is github.com/derekp7/snebu). For data retention, I keep daily backups for a couple weeks, weekly backups for 6 weeks, monthlys for a year, and yearlys for as long as I have space.

In Ask HN: How do you backup your servers?

derekp7·Feb 13, 2013

This post is so apropos for me (at least the title is) -- I'm perfectly fine with the Git part of github, but was struggling with the other parts of it. For example, trying to figure out how to make a versioned release. The download link they provide gets you the latest version from the repository. Took me a bit of searching and looking at other projects to realize that if you tag a release, it can make that available too. But now trying to get that to show up in the project page (using their auto page generator) -- I ended up having to hand edit the html code, because I wanted the web page to have a list of specific releases. But even their source download link isn't fully intuitive. For example, mine shows up as "https://github.com/derekp7/snebu/archive/1.0.tar.gz, but it downloads a file called "snebu-1.0.tar.gz". Oh, and to get t

Read the rest

he readme file to utilize their markdown, you have to call it reamde.md, instead of readme.txt. And I'm looking for where my users can leave feedback or discuss things, other then opening an issue in the issue tracker. But they don't have any mailing lists or forums (I guess most projects set up a separate forum, and link to it from the readme.md file or the gh-pages page). What I'd like to see is a document going over the entire lifecycle of all the features offered through github, including documenting the standard practices for items that they don't provide (such as the forums thing). Anyone got any pointers?

In How the Heck Do I Use GitHub?

Showing the 15 most recent of 16 sources. Enable JavaScript to search and browse every indexed mention.