1.0 Introducing Niente
Niente is the all-new FLAC integrity checker for Linux. It replaces and improves on the functionality of the Absolutely Baching FLAC Checker (AFC, as it was known).
Like AFC, Niente scans a folder structure full of FLACs and computes a digital 'fingerprint' of the audio signal in each FLAC found. If that fingerprint matches the one written into the file by the FLAC encoder at the time of its creation, the file is known to be unchanged since the day it first hit the hard disk. If the fingerprints are different, the file has suffered internal corruption or bit-rot of some sort, and you can get Niente to report to you about these failing FLACs.
Also like AFC, Niente will perform several logical checks on the tags in your FLACs. Where the ALBUM contains a recording year, for example, that's checked against the actual recording year stored in the DATE tag: the two should agree and Niente will tell you if they don't. Likewise, all the performers on a recording should be listed in the COMMENT tag, and the principle performer should also make an appearance in the PERFORMER tag: Niente will tell you if the two tags mention the same artist correctly or not.
Unlike AFC, however, Niente will do all this checking without ever once altering the FLAC files themselves. AFC used to update the FLAC directly, writing a new date into the TAGDATE tag, so that it could then work out how recently the last FLAC check had been performed when it met the same FLAC in a subsequent run. There's no fundamental problem with creating or updating a new bit of metadata like that -but it has the unfortunate side-effect of making the file appear 'newly updated' to any backup mechanism that concerns itself with the date a file was last modified and takes fresh backups of anything it detects to be 'newer' than the date of the last backup. Run AFC once a month, for example, and my entire 1.7TB music collection would be marked as 'brand new', and that would trigger a fresh backup to my NAS.
I could live with that (and did, for over 2 years!), because the NAS is in the loft and it doesn't take that long to copy the music from my desktop PC to the loft, over gigabit ethernet.
The trouble is, however, that I have recently invested in a personal Backblaze account (thoroughly recommended, by the way: completely unlimited cloud storage for US$60 per year. It's the best cloud storage deal I've ever come across!) to provide an 'off-site' backup of the NAS, up in the cloud. My Internet has the heady maximum upload speed of around 15Mbps, so you can imagine that having to upload 1.7TB to the cloud afresh every month would basically mean my entire upload bandwidth was permanently consumed by the backup! The days of updating the FLAC files to record the date of their last integrity check had, therefore, to be ended.
Niente differs from AFC in this one, crucial respect, therefore: when it scans a FLAC file, it records the date of the check and all the internal details it discovers about the file in a relational database (a Sqlite3 one, so it's free, open source and incredibly low on resources). The FLAC file itself never gets modified, so checking it doesn't cause a cascade of backups, one of which would block the Internet for a month!
2.0 Implications for AFC
Niente's release means that AFC as a product is now discontinued. You can still download it and install it, if you insist, from the usual places. But the script is now effectively 'abandonware' and will receive no further updates from me from July 1st 2021 on. You are accordingly strongly recommended to download and install Niente instead and to use that in the future. A message to this effect will appear if you ever upgrade to the terminal release of AFC.
Installation of Niente is achieved most simply by issuing the following three commands:
cd wget https://absolutelybaching.com/abc_installer bash abc_installer.sh --niente
The installation process requires access to the Internet and will prompt you for sudo privileges in order to be able to copy files to the /usr/bin folder.
3.0 Product Documentation
The user manual for Niente, which covers (I hope) all aspects of the program's installation, usage and reporting capabilities is available as a PDF download here. (3MB downloads).
If you prefer reading your manuals in an E-reader of some kind, the same documentation is available as an 85MB download in Epub format here.
4.0 Quick Start Guide
After you have installed Niente, it is suggested you run it once with the bare command:
niente
That should produce a page of help text; if instead it prompts you to install other software packages, that's just it getting its dependencies sorted. Once you've got those installed, try running Niente with the bare command once more. Once you get the page of help text, you know it will work.
You are then required to create a database and point it at the physical folders where your music collection resides. The command would look something like this:
niente --createdb --dbname=main_collection --musicdir=/sourcedata/music/classical
That will create a database called "main_collection" and then scan the /sourcedata/music/classical folder for any FLAC files lurking within that folder hierarchy. The results of the scan are stored within the database. Niente now knows where you music files can be found. The initial scan is fairly quick (about 30 minutes to scan my 1.7TB collection, for example), since it doesn't do a lot other than collect file names and locations.
At some point, you make Niente do real work with the command:
niente --integritycheck --dbname=main_collection
That will take potentially hours to do its work (for my collection, about 6 or 7 hours, in fact). It is during this process that Niente checks the metadata tags and physical properties of the audio stream, writing the results of its checks back to its database.
You then query the database once the integrity check is complete, with a command such as:
niente --report=screen --dbname=main_collection
That will produce three reports on-screen in sequence (press 'q' to quit out of each one and get the next displayed, until the last one quits and just dumps you back at the command prompt): 1) the physical integrity check; 2) the logical check of DATES/ALBUM tags; 3) the logical check of PERFORMER/COMMENT tags.
After your initial run, it is suggested that you schedule the database refresh process nightly and the integrity check weekly, using your crontab, with entries such as:
# Nightly refresh of the Niente database, every day bar Sunday # ----------------------------------------------------------- 0 3 * * MON-SAT /usr/bin/niente.sh --refreshdb --dbname=main_collection --musicdir=/sourcedata/music/classical # Weekly integrity checks of the music source, every Sunday at 1AM # ---------------------------------------------------------------- 0 1 * * SUN /usr/bin/niente.sh --integritycheck --dbname=main_collection
The heavy-duty integrity check so scheduled will not actually end up doing much work, because by default it is differential: it only actually scans files which were last scanned more than 27 days ago (by default). So, on the fourth Sunday after you run it for the first time, this sort of crontab will mean the entire collection gets deep-scanned once more, but for the other three Sundays, only those files new to your music collection will actually get deep-scanned, with the others being simply skipped past relatively quickly.
5.0 Product Iconography
The rather ghoulish image that heads this page is a photograph of the 'Cheshire Cat' sculpture within Christ Church Cathedral, Oxford -where one Lewis Carroll was a Mathematical Lecturer in the mid-Nineteenth Century. The Cheshire Cat in all his forms is Niente's 'mascot', because it is hoped the program will gradually fade away into the background, as it does it's import integrity checking job without making a nuisance of itself as far as cascading backups are concerned! Hopefully, all that should be left after you start using it is a slight smile of satisfaction that you know your FLAC files are fundamentally sound.