1.0 Introduction
Niente's main program display, when you finally get it running, appears generally as follows:
I say it 'generally' looks like this: the specific option numbers may change over time, as new functionality is added to the program, or old options are retired. Regardless of the specifics, however, the program display will always consist of three distinct areas, as follows:
At the top of the screen, the Header Area tells you what version of Niente you're running: as new releases of the software are developed, you'll want to make sure you're running on the latest by checking your program version here..
The Main Area of the screen consists of a series of menu options, each option performing one specific function. Broadly speaking, there are four main categories of menu option:
- Creation and maintenance of a database of what music files exist
- Performing various types of integrity checks on the music files mentioned in the database
- Reporting on the results of those integrity checks
- Performing various administrative tasks which affect the way Niente works
I'll briefly go through each of these categories now, though the detailed explanation of what each does can be found elsewhere in this documentation set.
Finally, the Footer Area is where various messages and progress indicators will be displayed, and where you type numbers to select particular menu options.
2.0 Database Creation and Maintenance
Niente's entire workflow is:
- Find FLACs on your file system
- Create a database storing details of those FLACs
- Use the database details to analyse those FLACs
- Generate reports from the database about which FLACs are corrupt or have logically inconsistent tag data
...and you can probably tell that having a database is pretty much central to making all of that work!
Your first order of business, having finally gotten Niente to work at all therefore, is to create a new database. You do that in three stages. First, you use Main Menu Option 9 to configure the DBNAME parameter to have an appropriate value. If you prefer, you can also simply open Niente's persistent configuration file in any text editor you like: it's in the $HOME/.local/share/niente folder and is called niente.conf. When you use option 9 to edit the same file, Niente will use whatever text editor your EDITOR environment variable has been set to. In a lot of distros these days, that defaults to nano, which is easier to use (for non-geeks, at least) than the other common one, vi. Here's me editing my configuration file using the option 9 editor:
As you can see, I've altered my DBNAME parameter to have a value of 'hjr' on this occasion. From the text above the parameter, you can also tell that the default database name (the one you'd get out-of-the-box, if you did nothing to the configuration file) is niente. Also note from that text: database names are case-sensitive, so 'hjr' would be a different database than 'HJR'.
Next, you tell the configuration file where your music collection lives, by editing the MUSICDIR parameter. You can see it listed in the above screenshot immediately below the DBNAME one and it's already been set to my /sourcedata/music/classical folder. Note that the folder structure you mention here must (a) already exist; and (b) live at the top of the folder hierarchy in which your FLAC files are stored, because Niente will visit every sub-folder below it, looking for FLAC files it can add to its database. In my case, for example, my music files all live on a NAS server in the loft and I map a local PC folder to them, resulting in a tree structure such as this:
Under the /sourcedata folder 'root', there are two sub-folders (audio and music). Within 'music', there's a 'classical' folder, within which are further sub-folders (A, B, C and so on), each one containing yet more sub-folders that represent a single composer. Inside every one of those composer folders are 'genre' sub-folders (so ballets are kept separate from choral works and so on). Inside a 'genre' folder, each individual recording will be stored within its own, uniquely-named folder. So, as you can see: there's a lot of levels to descend to before you finally find a FLAC! Never mind: so long as I point Niente to the top level in that convoluted folder structure, it will find all the FLACs stored beneath it. In my case, therefore, I need to set MUSICDIR to be /sourcedata/music/classical in order to pick up all my FLACs, wherever they may be hiding below!
So, once you've set your DBNAME and MUSICDIR parameters correctly, you can press Ctrl+X to exit the editor, then tap 'y' to confirm the save of the altered file, and finally press [Enter].
The modified configuration file is immediately applied to the running Niente session, so you can now move on to the second step: taking main menu Option 1 to actually create the physical database. As soon as you do, the screen will display the following sort of message:
The database creation process is over in mere milliseconds -because only an empty database structure is created at this point. As the message hints, the database name itself and the music folder root that you specified in the configuration file are 'hard coded' or baked into the database ftom the moment it is created. You cannot alter the database name or the music folder it specifies after the creation of the database. If you wanted to do so, you'd delete the database, modify the parameter file and create a new database entirely. The database exists as a single file called hjr.db (at least, in my case!), again stored in the $HOME/.local/share/niente folder, so deleting it is simply of matter of visiting that folder in your standard file manager, locating that file and doing the necessary.
Pressing a key when prompted by the above screenshot's message triggers Niente's hunt for music files:
The search process discovers what FLACs exist: that can take quite a long time if your music collection is large. Once the location of the FLACs has been determined, the display changes once more to display this:
The count-through of the discovered FLAC files is usually reasonably quick, since the Niente database is, at this stage, only being populated with the details of the files' locations. No great analysis of each file's tags and other metadata is being performed at this point, so all Niente has to do is, at is says on the tin: ensure that each discovered FLAC has had its precise folder location and file name captured correctly. Be patient as the record counter ticks ever-upwards in the footer area of the program display!
When all FLACs have been verified, the program display changes to this sort of thing:
You can then do as the last line of text tells you to do: press a key to continue: you'll then be taken back to the main menu screen. The database creation and initial population processes are now complete
From the above description of the database creation process, you can hopefully see that a new Niente database starts off as being nothing more than a snapshot of what FLACs exist at the moment of database creation. If you were now to add new recordings to your collection or modify the details of any existing ones, the Niente database will not know about it because those changes were done after the initial poopulation of the database with recording details. However, any time you perform one of the possible integrity checks that Niente can do (which I'll come on to shortly), Niente always has another scan of the file system and thus picks up new and changed files: integrity checks therefore trigger the same scan-and-load process you've just witnessed as part of the database creation process.
Every time you perform a create or refresh database operation in Niente, a log of that operation is created (viewable by taking Main Menu options 8 -> 14):
You can clear down this log at any time by taking Main Menu Option 11.
Should you ever wish to wipe the entire contents of the Niente database, you can do so by taking Main Menu Option 2. This deletes all records of which FLACs exist but without destroying the database internal structures. It also, of course, does not touch the FLAC files themselves, so they are left quite safe! When you next run a full or differential integrity check Niente will re-populate the various database tables from scratch, after which normal program running is resumed. There should never really be a need to take this option, but it's there if you think you need to use it.
3.0 Integrity Checking
If you were to use a tool such as the DB Browser for SQLite to look at the contents of the Niente database just after it had been first created, you see something like this:
That tells you how little work a database create or refresh operation actually performs! It certainly knows what FLACs exist, but all the 'null' columns shown in that screenshot tell you that that's all Niente knows about. For it to know a FLAC's bit-depth or its date, performer, or comment tag data, it has to visit each listed FLAC in turn and extract that information from the file itself and populate the table columns you see are listed as being blank here. That analysis process is termed an integrity check and it has to be performed as a separate exercise from a database creation or refresh process, by taking Main Menu Options 3, 4 or 5.
The three different integrity check options can be distinguished from each other as follows:
- Full: analyses every FLAC listed in the database, whether recently checked or not. Both physical and logical checks are performed.
- Differential: analyses only those FLACs which are listed in the database as suffering from either physical or logical errors. Both physical and logical checks are performed
- Fast: analyses only those FLACs which are listed in the database as suffering from either physical or logical errors. Only logical checks are performed
See the sections of the user manual regarding the nature of physical integrity checks and logical inconsistency checks.
Note that not having any tag data stored in the database (which, as we've just seen, is true for every FLAC immediately after database creation) is regarded as a logical error of sorts -so, all three types of integrity check will end up analysing every single FLAC if run immediately after initial database creation, regardless of the fact that they are different 'types' of check. There won't, in other words, be a difference betweena full and differential check at this point, since the differential check will have to check every discovered music file, just as a full check would do. Only on second and subsequent checks will the difference kick in. It is true, however, that whilst the fast check will also have to visit every music file listed in the database, it will be significantly quicker to complete than the other two, because (a) it doesn't perform an initial re-scan of your file system to try and pick up new and changed FLACs; and (b) it skips the crucial business of 'fingerprinting' the audio signal in your FLAC files and thereby assessing its internal, physical integrity. I would not particularly recommend skipping that physical check on your first run, though: stick to taking menu options 3 or 4, for now.
Here's the general appearance of any integrity check:
You can see that each FLAC is being visited in turn and its metadata tags are being inspected for details such as album name, composer name, genre, recording year and so on. That's actually a screenshot of a fast integrity check; here's the Full equivalent:
You'll notice that the only substantive difference between them is that last item of data shown for each recording, the 'MD5 Hash' is there for a full (or differential) check, but it's missing from the fast check. The 'hash value' shown is the physical 'fingerprint' of the audio signal contained within a FLAC file which Niente uses to determine whether internal corruption of that signal has begun to take place. Performing that sort of inventory of the audio signal takes quite a bit of time (imaging having to 'fingerprint' the entire Ring Cycle, for example!), which is why the fast check is so much, er...., faster than the other two types of check!
Once this detailed analysis process has completed, you'll be prompted to 'press a key to continue' once more and then you'll be returned to the main menu. Nothing will appear to have happened, particularly! However, if you were able to look at the Niente database directly once more, you'd see a considerable change from before:
Now you can see that all the columns of data for each FLAC file have been filled in with meaningful details: that which started out blank has now been populated with data extracted from the FLAC files themselves. More importantly from Niente's perspective, the fact that the data is now present in its database means that it's now capable of producing reports on that data, which you do by taking one of the many reporting options hidden behind Main Menu Option 8. It is by running reports that you find out what exactly Niente discovered during its last integrity check.
Before I get on to discussing those reports, however, I should briefly mention before that there are two further types of FLAC check that Niente can perform for you, available by taking main menu Options 6 and 7.
Option 6's Album Art Checkup function assumes (as per Axiom 16) that you will have embedded album art within each FLAC. This option will (a) ensure that this is actually so, and that you haven't accidentally forgotten to do this; and (b) stores measurements of the embedded album art it finds (as in, pixel height and width). You will later be able to produce reports on which files are missing album art entirely, have album art which is thought too small or pointlessly too big or have album art which is not perfectly square. A full integrity check automatically performs this album art check as part of its routine operations; a differential integrity check does not, which is why Option 7 allows you to invoke it as a separate check on its own.
Option 7's Check for potential volume boosts is a completely separate analysis (that not even a full integrity check performs) about the audio signal peak volume levels contained within each FLAC. No adjustment to the volume of any FLAC is actually performed: the check simply records what the maximum volume boost of any FLAC could be, without introducing distortion. Once the analysis is complete (which can take quite a long time, which is why this check is not included in a standard full integrity check), you can run reports to show which files could benefit from a volume boost. The actual volume boosting process can then be run at your leisure by running my Semplice program inside the relevant folders.
4.0 Reporting
Performing an integrity check merely collects data, as we've seen. In order to learn anything useful about the data that's been collected, you need to take Niente's main menu Option 8 to produce a variety of reports:
As you can see, there are three main 'groupings' of reports, with perhaps the most important being the 'General Reports' one.
Starting with Reports Option 1 (the 'quick aggregate statistics' one), then:
You see here how each report pops up in a separate window, distinct from the main program menu running in the background. You can close the report window either by clicking its 'X' icon in its title bar, or by pressing the letter 'q' (to 'quit'). If a particular report is long, you can scroll up and down through it at will, just by pressing PgUp and PgDn (to jump multiple lines at a time), or by pressing the up and down cursor keys (to scroll one line at a time).
The quick aggregate statistics report is handy for showing you (a) when the last time an integrity check was performed (see Statistic 9) and whether or not any discovered FLACs have fallen foul of any of the integrity 'rules' (which are explained in detail in the physical corruption and logical inconsistencies sections of this manual. If you see a non-zero number for any of the listed statistics, you can then run the detailed report relating to that statisic. In my case, for example, I appear to have 41 'inconsistent performer/album tags': I can therefore switch back to the reporting menu and take Option 7 to get a detailed breakdown of which files are causing grief:
Here you can see that in one of my Aaron Copland recordings, I've declared the 'distinguishing artist' component of the ALBUM tag to be 'Hampson', but then gone on to write 'Hugh Wolff' into the PERFORMER tag: by Axioms 5 and 6, that sort of discrepancy isn't supposed to happen. Now: I can't know, just by looking at this report, which is the "correct" name to list in either tag -and if I can't do it, neither can Niente, which is why Niente can't and won't fix up anything it reports on as being 'wrong'. These reports are, in other words, simply a set of instructions to you, yourself, to visit each FLAC shown in turn and work out how to re-tag (or re-rip) the relevant files.
When you've worked through all the listed errors, you can run a new integrity check to pick up the changes and re-tagging activity -and a re-run of the quick aggregate statstics report should now show the relevant statistic with a zero row count.
As you can see, therefore, Niente helps you keep on top of silent, physical corruption and the sorts of logical inconsistencies that arise from mistaken tagging. It will never fix such issues for you, but it points you to where problems have arisen, so that you can take whatever the appropriate fixup action might be.
5.0 Administration
The last section of Niente's main menu options are all to do with fairly mundane house-keeping activities! Chances are, once you've got Niente working reliably as you wish, you won't need to use these options very often at all.
As already mentioned, there are various parameters that control how Niente looks and behaves, set in the persistent configuration file. That file is a simple text file, called $HOME/.local/share/niente/niente.conf. You can edit it at any time with your choice of favourite text editor. To save you having to navigate your own way to the Niente configuration folder, though, Niente's main menu Option 9 lets you edit the file in-place. Taking that option will open the file in whatever your operating system has set as the default text editor. Many distros make this vi, but quite a few these days set it to be nero. You can always change the system default editor by setting EDITOR=xxx in your .bashrc, where 'xxx' is the name of the editor you prefer. Should no default editor be specified by your operating system, Niente will attempt to open the configuration file in nano (which is a bit more naiive user-friendly than vi, I think!)
Once opened, you can edit the text at will and save the changes in whatever standard way your text editor provides (in nano, for example, you'd press Ctrl+X, then Y to save). Be warned, as the file itself tells you, do not, ever, edit the text contained within the double-lined rectangle forming the first ten lines of the file. If you do and cannot revert back to the previous unedited version, you will have corrupted the configuration file and Niente will not restart. Instead, you'll get this sort of error message:
The fix here is simple, but tedious: just delete the existing (and now corrupt) configuration file (by running the command rm $HOME/.local/share/niente/niente.conf) and then re-run Niente as normal. That re-run will trigger a warning about 'no config file found, so I'll download a new default one for you', and once that download has been allowed to complete, you'll be back in business. The new default configuration file will, of course, have obliterated any customisations you'd made to the old one -so it's best not to edit that part of the file and corrupt it in the first place!
Finally, when Niente is scanning and analysing FLAC files, it generates a lot of 'working data' that it stores in various temporary files. If a second Niente session were to be launched whilst the scan or integrity check was taking place, the new session would over-write the temporary files being used by the first. The result would be a logical mess (and a corrupted database, probably). Accordingly, file scans and integrity checks take out a 'program lock' as they are first launched, preventing a second session from being able to perform those operations simultaneously. When you then quit the first Niente session, it will clear out the lock it took, and the second session could then begin to perform scans or integrity checks. The only fly in this particular ointment is if you accidentally kill Niente (perhaps by shutting down the terminal session it's running in by clicking its 'X' icon in the title bar), or your PC crashes or unexpectedly powers down in the middle of a FLAC scan or integrity check -for then, the Niente session never gets a chance to remove the lock it originally took. When your PC comes back and you try to re-run one of these options you'd merely see a message such as:
The left-over lock makes the new Niente session think an already-running session exists and you'll be prevented from running such operations in the future. If you are certain the detected program lock is spurious, because of a program or PC crash for example, then you can simply delete the lock file. You could do this manually (the lock is just a file called $HOME/.local/share/niente/nientelock), but to save you having to navigate your way through a deep file structure, Niente main menu Option 10 will remove the lock for you. Do not remove a program lock unless you know absolutely that it is safe to do so.
Main Menu Option 12 displays the GPLv2 software license under which Niente is released. It is a requirement of the GPLv2 that users of software licensed under it should easily be able to obtain a copy of the license text: option 12 makes that happen.
Finally, Main Menu Option 13 wipes the contents of the Niente operations log (which records every time a database is created, integrity checked and so on). There's no great reason for ever doing this, but one could imagine that several years down the track, scrolling through records of the integrity checks you ran three years ago might be more tedious than it's worth! This option therefore zeroes the operations log, without that affecting any other part of the Niente database.
6.0 Automation
In all the above, you've been using Niente's functionality interactively, via the program main display and the business of taking numbered menu options. Some of Niente's key tasks, however, are more meant to be run automatically, without user intervention. For these tasks, Niente can be invoked from the command line with a variety of runtime switches. These are keywords preceded by a double-hypehn. You add a runtime switch to the basic 'niente' command to run Niente in that specific mode. The following switches are available:
- The --check-full runtime switch triggers a full database refresh plus both physical and logical integrity checks . It is the equivalent of taking Main Menu Option 3.
- The --check-differential runtime switch triggers a full database refresh, followed by a physical and logical integrity check of only those FLACs previously found to have been suffering from physical and/or logical consistency problems. It is therefore the equivalent of taking Main Menu Options 3.
- The --check-fast runtime switch triggers a logical-only integrity check of those FLACs previously found to have been suffering from physical and/or logical consistency problems. It is therefore the equivalent of taking Main Menu Option 5.
- The --check-art runtime switch triggers a re-scan of all FLACs already present in the database for an analysis of their album art properties. It does not refresh the database before it gets to work and performs no other physical or logical checks. It is the equivalent of taking Main Menu Option 6.
- The --check-volume runtime switch triggers a re-scan of all FLACs already present in the database for an analysis of their peak volume levels, to determine whether any could take a volume boost without introducting distortion. It doesn't refresh the database before running and performs no other physical or logical checks. It is the equivalent of taking Main Menu Option 7.
- The --checkver runtime switch has no counterpart in the program's normal main menu interface, but triggers a check of the Absolutely Baching servers to see if a newer version of the software is available to download and install. If one is found, you are prompted to provide a sudo password to allow the installation of the upgraded version (but you can always decline to upgrade if you prefer).
Using these parameters, for example, you can schedule full and differential integrity checks to take place regularly via crontab (or similar), so that your Niente database is always kept up-to-date and FLAC analysis is performed frequently enough that you can spot potential problems before they get too serious and widespread. My own crontab, for example, has these entries:
# Twice-weekly differential check of the music files with Niente
# ----------------------------------------------------------------
59 22 * * SUN,WED /usr/bin/niente.sh --check-differential
# Complete integrity check every quarter
# ----------------------------------------------------------------
0 3 1 1,4,7,10 * /usr/bin/niente.sh --check-full
As you can probably work out, I am performing a differential integrity check twice a week, on Sundays and Wednesdays, at about 11pm. That ensures that Niente becomes aware of any new acquisitions to my collection within about three days of them having been made. Additionally, on the first day of January, April, July and October, at 3AM, I perform a complete integrity check of all my FLACs. In this way, I aim to keep the amount of work Niente does on a routine basis to a minimum, but still ensure that it re-checks files it has already checked frequently enough that backups of known-good FLACs are still available in the event of recovery from backup being required to fix freshly-detected physical file corruption.
7.0 Summary
That hopefully all gives you a sense of what Niente does, how it does it and the order in which it should all be done -and how you can automate the heavy-lifting parts of the process so they happen at night or when you're otherwise not wanting to be bothered by it!
Broadly speaking, you use Niente to:
- Create a database as a one-off exercise
- Do regular differential integrity checks
- Do occasional complete integrity checks
- Run quick aggregate statistics reports about once a week -and run appropriate detailed reports should anything be listed on that quick aggregate statistics report as being of concern
By automating the 2nd and 3rd of those tasks, too, you can make Niente do what it was always intended to do: disappear gently into the background, whilst yet managing to ensure your FLACs remain physically healthy and tagged in a logically consistent manner.
|[Back to Front Page]|[Physical Corruption]|[Logical Inconsistency]|