After reading some scare stories about the risks of committing all your work to a proprietary database, the Lightroom catalogue, I decided to put together a few notes on how I minimise that risk.
Catalogue size: If your collection of images to be catalogued is over the 50,000 you may find it better to create more than one catalogue from a performance perspective. Though I have found the greater impact on performance while browsing the catalogue is the individual image file sizes. Creating and viewing 1:1 previews of very large and complex Photoshop/TIFF files does take some time and can slow down the browsing process considerably. When I say very large and complex I mean files in excess of 500MB with multiple layers. I do a lot of high resolution scanning for restoration and reconstruction jobs and have files that during work can reach ~2GB, I don't include these 'in progress' files in the catalogue but will export flattened JPG files for inclusion in the catalogue as a reminder of where I am with a job.
If possible put your catalogue and scratch/cache folders on an SSD, this helps performance quite a bit as the preview files are kept in a sub folder of the catalogue location.
When you create a new catalogue there are a few setting to consider before you start importing your images.
In Catalogue Settings under the General tab choose to back-up the catalogue every time Lightroom exits. When you exit Lightroom it will ask for the location of the backup and confirmation that you would like to run the backup, so if you haven't done any/much editing you can always click the skip this time button. The first time you choose to run the backup have Lightroom create the backup on a different disk to the catalogue, also tick the Test Integrity and Optimise Catalogue boxes, Lightroom will remember these preferences.
On the File Handling tab in Catalogue settings choose the smallest Standard Preview size and Medium quality to begin with, this will help to minimise the amount of space used for previews and will help a little with performance, you can always go back and increase these settings later if you desire. The setting for discarding 1:1 previews is a bit of a compromise between storage and performance. Discarding 1:1 previews after one day will significantly cut down on the disk storage required for previews but if you open your catalogue every two or three days to work on a set of images then LR will have to rebuild the 1:1 previews every time, impacting on performance. On the other hand never discarding 1:1 previews will result in a very large preview cache so you'll have to work out for yourself what setting works best with your pattern of use.
Now the important bit, on the Metadata tab in Catalogue Settings there are two settings that are of significance: 'Include Develop settings in metadata in JPEG, TIFF, PNG and PSD files' and 'Automatically write changes into XMP'. The first of these setting is very useful as it includes everything you do in the Develop module within the metadata in the listed file types, so if you import them into another LR catalogue or open them with ACR/Bridge you should see all the edits you made. This only works for the listed file types as Adobe are understandably reluctant to start writing extensive metadata into proprietary RAW files. The second option automatically writes all changes to an external XMP file with the same name as the associated image file.
I have no hesitation in suggesting you tick the 'Include Develop settings...', it's useful if you edit in both LR and PS (and anything else that’s capable of reading the develop metadata) and it means if everything goes irrecoverably wrong you can import the images into a new LR catalogue and you'll see all the edits and any other metadata that was added such as copyright info and keywords.
The option to 'Automatically write changes into XMP' effectively does the same for all file types, including proprietary RAW, it's able to do so because original proprietary files are not modified directly, only the associated XMP file. Be aware though that this setting has been reported as causing performance issues for some people. The evidence is rather vague as to the circumstances and it's not an issue I've seen myself so I can't offer any specific mitigation if you are one of those who suffers a performance hit. My advice would be to turn the option on when you create the catalogue, you can always disable it later if you run into performance problems.
If you do need to turn it off for performance reasons all is not lost as you can tell LR to create/update the XMP files as and when you need to. If you have been working on an image you just need to press CTRL+S or go to the Metadata menu and choose 'Write metadata to file'. This creates/updates the XMP file for all images that are selected so if you have been working on a number of files in a folder or collection of files you can select all the files in that folder/collection and hit CTRL+S and the changes will be saves for all the files that have been edited.
Another option would be to copy and convert all your RAW files to DNG as you import them so all metadata and develop settings can be maintained within the files rather than in XMP files, but if you have a large collection of RAW files this may not be practical, and you may prefer to work from your cameras native RAW format, especially if you also use the RAW software that came with your camera.
Once you have created your catalogue and made the appropriate settings it's time to import your images. If you have an existing collection of files in a folder hierarchy that you are comfortable with you can import them without moving or copying them, just select the 'Add from current location' option at the top of the import dialogue. Just remember that if you want to move things around it's better to do that in LR rather than at the file system. If you do move things around in the file system and you are using XMP files remember to keep the XMP files with the image files. You can fix the issues caused moving things in the file system without too much difficulty but I'll leave that for now.
If you follow the above you have a pretty robust LR setup. You have backups of the LR database on a separate drive, each one tested for integrity, and you have all the important edits and metadata stored independently of the database so if all else fails you can create a new LR catalogue and import the image files into that. The structure of your folders won't change assuming you added the files without moving/copying them and LR will automatically read the embedded info for the compatible file types and the associated XMP files for the proprietary RAW files.
If you suffer issues with a sub-set of the images in a LR catalogue (not something I've ever seen, but let’s say just the sake of argument) you can delete the individual files/folders from your working catalogue and use the 'import from a Lightroom catalogue' to import those files/folders from a catalogue backup.
Another advantage of keeping all the edits embedded and in XMP files is that you can have image files in more than one LR catalogue and changes made in one catalogue can be synched to the other by selecting the images and choosing the 'Read metadata from file' option.
Finally, if your metadata/edits are embedded and/or included in XMP files then all your image file backups will include a snapshot of the image at that moment in time. This can be useful or a hindrance depending on the circumstances but if you have recovered image files that are referenced in a LR catalogue then LR will warn you that the embedded/XMP data is different from the catalogue data and will give you the choice of updating the catalogue with the image data or updating the image data with what’s stored in the catalogue.
As far as my testing goes the only significant thing you will lose if a catalogue becomes completely unusable/irrecoverable is the edit history, but I'm not aware of any application that would provide a mechanism for rebuilding the edit history.
I hope this helps!
All the best, Adrian