Niente Version 4 : Database Creation and Population

1.0 Introduction

Niente is a database-driven program. It requires that a database exists which records the existence of all the FLACs in your music collection. Apart from the initial discovery of these FLACs, Niente does not do file system visits or scans: when it's producing a report, for example, it's obtaining all the data in the report from its database, not by taking a fresh look at the FLACs directly. Similarly, when Niente performs a new integrity check, it's reading the FLACs it knows about because they're listed in its database: it's not doing an on-the-fly scan for FLACs, but simply reading the contents of the ones the database tells it to visit. Without a database populated with details of which FLACs exist where, therefore, Niente can do nothing useful!

It's therefore important that, when you first run Niente, you immediately 1) create a fresh database and 2) populate that database with details of which FLACs exist in your music collection. That's why the Database menu is the one you always open Niente at when it is first launched.

2.0 Database Structure

I'll just pause at this point to explain the structure of the database that Niente uses. The main table is called TRACKS and that lists each FLAC filename, along with 13 attributes or characteristics concerning each file, such as its PERFORMER tag, or what bit-depth the music in a FLAC was recorded at, or what the COMPOSER tag is set to, and so on. If you have 1000 individual FLAC files in your music collection, the Niente database will contain 1000 rows in the TRACKS table, one row per file. If you've got a single Beethoven symphony, comprised of 4 individual files, there'll be four rows in the TRACKS table to describe each of them individually.

In addition to TRACKS, there is a MAXVOLUMES table. This will list each folder of FLACs, rather than each FLAC file individually, and for each folder, the peak loudness of all the FLACs contained therein is recorded. So, say you have those 4 FLACs representing a Beethoven symphony: one is at -0.3dB from maximum loudness, the second is at -2.3db, the third is at -0.8dB and the fourth is at -1.2dB. This means that the maximum volume boost you could apply to all the files in that folder without introducing distortion is +0.3dB: that would boost file 1 to be at 0dB, the theoretical, non-distorting maximum possible volume.  In the MAXVOLUMES table, therefore, there will be just one row for the whole of (say) 'Symphony No. 5', and the peak loudness will be stored as -0.3dB, which is the largest possible volume boost that could be applied.

Finally, the Niente database contains another table called ALBUMART. This contains one row per FLAC file once more, and against each row is stored the horizontal and vertical dimensions (in pixels) for the album artwork that has been embedded within a FLAC, if any. This data then allows Niente to report on non-square, over-large or under-sized album art.

The TRACKS table is populated by performing a Load or an Add/Remove operation, using options 2 or 3 from the Database menu. Doing so is extremely quick, even on large music collections, since the only data being collected is the full path and filename of each FLAC file in your collection. Filling out the 13 characteristics or attributes of each FLAC is what an integrity check will do -and those take a long time to complete for large music collections.

The ALBUMART and MAXVOLUMES tables are only populated by performing an album art or volume boost integrity check, using options 4, 5 or 6 from the Integrity Checks menu. These checks again take a long time to complete for large music collections. They each derive the information about what FLACs to interrogate from the contents of the TRACKS table, so if that table is empty, no album art or volume level checks can take place.

From all of that, I hope you can see that TRACKS is the 'driving table': its contents are what tell all integrity checks are the files to scan and what, therefore, can subsequently be reported upon.

3.0 Creating the Database

Creating a Niente database is a matter of just a second or two: it takes no time to create a file in which a bunch of empty tables exist, waiting for subsequent data entry. It's done by taking the Database menu, Option 1: Create a music database:

You are asked to provide a name for the database. I'd suggest keeping it short and sweet and without spaces inside the name (use underscores, as I've done in the above screenshot ,where you'd otherwise be tempted to type a space), though you can in practice use spaces and have as long a database name as you'd like. Long names will get variously truncated on subsequent screen displays, though the full name still gets used correctly internally.

You cannot leave the database name blank: you'll be told off if you try and be given another chance to type something sensible. If you type in a name of a database that already exists, that will also trigger the display of an error message, after which you'll again be given another chance to type something meaningful: Niente simply won't over-write an already-existing database.

If you provide a never-before encountered, not-null database name then, in a matter of a second or two, a file of that name plus a ".db" extension is created for you in the $HOME/.local/share/niente4/db folder, where all Niente databases live and a confirmation message is displayed:

Note the advice that the new database needs populating, which requires a scan of your music folders. We'll get to that in a moment! For now, clicking OK (or pressing [Enter]) at this point returns you to the main menu.

You can create as many databases this way as you like. You might, for example, want separate databases for your jazz collection, distinct from your classical one; or maybe you want to keep your 19th Century music in a different database from your 20th Century stuff... I wouldn't particularly advise that approach, but you're certainly allowed to do it. Only one of your databases can be the 'default' database that Niente opens automatically every time it's launched, but you can tap the letter 'C' when in the database menu to change what the working database is at any time.

4.0 Setting a Default Database

Creating a Niente database results in this sort of thing happening on your file system:

Each new database is represented by a separate file -and all databases start off at about 40KB small, so they're trivial in size. Here, I have created three distinct databases: whenever you launch Niente in the future, you'll be asked to pick one to work with:

Each file stored in the $HOME/.local/share/niente4/db folder is listed on this form: your job is to up- and down-arrow until the one you want to open is highlighted and then hit the space bar to select it. An asterisk appears next to the name and you can then click OK or press [Enter] to confirm the choice.

At any time after launching Niente, too, you can tap the letter 'C' whilst in the Database menu and the same form will appear, allowing you to change which database is in-use for the remainder of your Niente session.

If you don't mind being prompted in this way every time you launch Niente, then fine: you don't need to do anything else. If you only have one database, though, or if there's one database that you use much more than any other, you may want to declare it the default database. You do that in the Administration menu, Option 1: Edit Configuration File:

You just arrow down to the 'Default music database name' entry and type in the name of the database you want to use by default: just type the database name, not the name of the physical file (in my example, the database name is 'main', the physical file is 'main.db' and I therefore am typing in the 'main' version, without the .db extension).

Once the modified configuration parameters have been submitted (click OK or press [Enter]), subsequent launches of Niente will open the named database automatically, without prompting. You can still tap 'C' on the Database menu to switch away from the default to another database, but program startup will, at least, have been streamlined.

The database in use at any given time is listed on the main program display, on the third line of text at the top, in yellow:

...in this case, the 'main' database has been opened and is in use.

Which files get integrity checked, or reported on, depends entirely on the contents of the database that is in use whenever those options are taken.

5.0 Populating a Database

Creating a database was, as you saw, a matter of mere seconds, because hardly any work is required to create something that's just 40KB on disk! It is an entirely separate matter to populate the database with records of what FLAC files exist in your music collection. That's something which is done by taking the Database menu, Option 2: Wipe & re-load a daatabase from scratch or Option 3: Add/Remove files incrementally.

The difference between the two options is that Option 2 wipes then populates, whereas Option 3 simply populates. In other words, if you've got a database you've been using for some time and it contains details on 15,701 recordings, taking Option 2 deletes all existing rows, scans the file system, finds 15,701 "new" files and adds them back to the database from scratch. Option 3, on the other hand, will scan the file system for files which are not already in the database and adds them to the existing 15,701 rows.

Option 3 will also delete from the database any rows which reference recordings which no longer exist in the chosen music folder. If you stored a FLAC in a folder called 'Syymmphiny No. 5' for example, then that would be in the database under that folder name; you spot the error and change the folder name to 'Symphony No. 5'... not only do you want 'Symphony No. 5' added to the database, but you also want the now non-existent 'Syymmphiny No. 5' removed from the database: Option 3 will do both of these things for you.

Of course, when you've just created a database, it starts out with no rows in it at all and therefore everything found on disk will 'look new' to Niente -meaning that, just after database creation, there's no actual, practical difference between the two options: Option 3 is going to find 15,701 'new' files, just as Option 2 would do. Once you've populated a database, however, Option 2 is not something you'd want to take very often, if ever: you've gone through the pain of collecting data on 15,701 recordings and don't really want to have to re-collect that data from scratch. If you simply want Niente to take note of the 5 new CDs you ripped and added to your collection yesterday, therefore, you'd take Option 3 to add the 5 new recordings to the existing database.

If it helps to think of this way: Option 2 is a 'wipe and full scan' and Option 3 is an 'incremental scan, with deletes as well as additions'.

Whichever option you choose, you'll have to tell Niente what folder structure to scan to discover new FLACs. This needs to be a folder that contains directly within itself FLACs, or sub-folders (or sub-sub-sub-folders!) that then contain FLACs. There need to be some FLACs somewhere inside the folder you mention at this point, though: you'll be warned if no FLACs are detected at all.

Be warned that you can't point one database at two different folder structures.

For example, suppose my folder structure is as follows:

I have a 'parent' folder called Music, within which are three distinct 'child' folders as shown. If I point Niente at /Music/classical, I cannot subsequently add FLACs found in the /Music/overflow folder to the same database. Doing so will certainly populate the database with recordings found in the /overflow folder but it will simultaneously delete from its database anything that was previously found in the /classical folder.

If I wanted both /classical and /overflow FLACs to be recorded in the same database, I'd have to point Niente at the 'parent' folder /Music: then it will read both child folders at the same time and FLACs in both 'branches' will be logged in the one database simultaneously. Of course, this would mean that my Niente database would also pick up on any FLACs stored in the /non-classical folder, which would probably cause grief down the road (Niente doesn't understand non-classical music very well!)

So, your music's physical organisation needs to be logical and well thought out. In my case, I actually only want Niente to scan for classical music in my main collection, so pointing it at /Music/classical will do just fine... but I can only do that and get away with it because I planned my folder structure sensibly!

Anyway, with all those warnings out of the way, I now take Databse menu, Option 2:

The dialog that now appears allows you to navigate around your file system until the 'root' of your music's folder structure is reached. The form always starts off in your home folder, which is why /home/hjr is the starting point in my case. You can use the top panel (the one where I've got the folder 'hjr' highlighted in red) to navigate in a point-and-click sort of way: pressing the Space Bar whilst a folder name is highlighted there takes you 'into' that folder. To go back up a folder, you can tab round to the second panel (the one showing /home/hjr in the above screenshot) and just start deleting the last part of the path displayed there: the top panel will change to reflect what's typed in the second panel. In fact, you may find it easiest to tab to the second panel and simply type out the required path by hand: the top panel will merely give visual confirmation that you've typed it correctly!

Be careful to actually 'enter' the folder you want to be the root of your scan. For example, here it looks as if I've selected the /classical sub-folder:

The word 'classical' is highlighted in red at the top, after all. But look closely at the path typed in the lower panel: the word is 'classic' which isn't correct at all. If I clicked OK (or pressed Enter) at this point, the scan would fail completely, because an invalid path had been submitted. What I need to do instead is to tab round to the top panel and press the space bar with the 'classical' item selected:

Now I know I'm 'in' the /classical sub-folder, because I can see all the sub-sub-folders that exist within it. I can therefore press [Enter] at this point, confident I've pointed Niente to the right place.

Immediately I submit the new path, Niente runs off to the file system at that location and starts scanning for FLACs, either directly within it, or within sub-, or sub-sub-sub-sub folders of that 'parent' folder:

You get a brief visual confirmation of the folder being scanned and the database the scan results will be loaded into. This may not appear for very long, however, as the scan for FLACs is relatively quick to do: remember, you're simply producing a list of file names at this point, not working out all their tagging internals and physical corruption status. So, very swiftly, the screen will change to something more like this:

You get confirmation of how many FLACs were found in that location, and can then click [OK] or press [Enter] to return to the main menu. And that's really all there is to populating your new database! It takes hardly any time to do, even on the largest of music collections.

6.0 Configuring a Default Music Folder

You've just seen that I've populated my database with knowledge about 15,701 FLACs. If I were to quit Niente at this point, rip some fresh CDs, relaunch Niente, and ask it to do a Database menu, Option 3 to add the new recordings to the database, I'd see this:

It's the same folder selection dialog we saw back in Section 5 when populating the database in the first time. Niente doesn't remember which music folder you pointed to between runs, in other words. That's not especially an issue: as you can see, it will happily prompt you to remind it every time you ask to do a new file scan. Nevertheless, it's not very convenient!

Fortunately, you can fix this issue by setting a default music folder in Niente's persistent configuration file, using the Administration menu Option 1:

You simply type in the correct, full path to the 'parent' music folder in the first of the parameter fields, then click [OK] or press [Enter] to submit the new value. No validation of your entry is made: Niente takes you at your word and if you type in an invalid path or folder name, it's not going to check your homework for you! The consequences will be a problem, though: if you type in an invalid path and then go and do a Database menu Option 3 to add new recordings to your database, you'll actually only achieve the wiping out of the existing records in that database! That's because Niente won't find new recordings in the default path (because it was type in incorrectly), so won't be able to add anything. But it will also not find any of the existing recordings in the default path (because it doesn't actually exist!) and so will delete them all.

So: make sure you type in a valid full path to your music collection, please!

Once you've set a correct default music folder in this way, you can take Database menu Options 2 or 3 without being prompted for a working music folder: the default is assumed to be the 'working' folder for this session. Should that ever not be true, you can tap the letter 'F' whilst in the Database menu and change the working music folder. So non-default databases and non-default music folders are always selectable, should you need to.

7.0 Wiping a Database

Database menu, Option 4: Wipe a music database shouldn't really be needed very often, if at all. It literally deletes all data out of the TRACKS, ALBUMART and MAXVOLUMES table (as ever, wiping the database has zero effects on the FLAC files themselves. Niente never deletes or modifies in any way FLACs it knows about).

Be warned that wiping a database takes place immediately and without warning (largely because a database can be re-populated by simply taking Option 2 again, so it's in a sense a non-destructive action). Though not harmful to any FLACs, an unintended wiping will mean having to re-scan and re-check large amounts of data, which can be a bit of a pain. So, be reluctant to use this option -and make sure you know what your currently-selected working database is before you take it, lest you inadvertently wipe the wrong one!

8.0 Conclusion

Niente databases are easily created, and relatively easily populated. Having created a database, it's sensible to set it as the default database.

Database creation is a distinctly different operation, however, from database population. For that you do a file system scan, either starting from scratch or merely adding to what's already in the database. File system scans need to be told where to start scanning -and that folder needs to be at the 'top' of a folder structure containing all the FLACs that are to be recorded in the database.

Whatever 'top' folder you choose to scan can be set as the default music folder, meaning that Niente will assume it should use that as its working music folder any time Niente's launched in the future.

Defaults can be over-ridden: tap C to change the working database; tap F to change the working music folder.

Populating a database is, in any event, a relatively swift operation since the data collected is very minimal: merely a record of the full path and filename of every FLAC discovered during the file system scan.


[Back to Font Page]|[Integrity Checks]|[Reporting]|