Post

ajfs - Index file hierarchies to be searched, compared, duplicates found, etc.

Posted
By André Jacobs
3 min read
ajfs - Index file hierarchies to be searched, compared, duplicates found, etc.

I just released my latest project ajfs (written in Go).

ajfs allows you to index (snapshot) a file system hierarchy to a single flat file database which can then be used in an independant and offline (as in on another machine) way to do the following:

  • Find duplicate files or entire duplicate subtrees.
  • Compare differences between databases (snapshots) and or file systems.
  • Find out which files would still need to be synced to another system.
  • Search for entries that match certain criteria.
  • List or export the entries to CSV, JSON or Hashdeep.
  • Display the entries as a tree.

See the documentation for more details.

Why build this yourself?

I have needed a tool like this for a number of years and never quite found something that meets all the criteria or that I would use consistently. Most of the time it would be cobbled toghether shell scripts and 3rd party tools that didn’t quite met my needs.

Usage

Please see the examples for more detailed example use cases.

The “root path” is the path to the file system hierarchy from which a snapshot is created from.

The following is just a couple of examples of what is possible with ajfs.

  • Create a new snapshot.

    1
2
3
4
5
6
7
8

      # create the default ./db.ajfs database and scan the specified path
  ajfs scan /media/backups/e-books

  # specify where the snapshot database should be stored
  ajfs scan ~/mybooks.ajfs /media/backups/e-books

  # calculate file signature hashes and show progress updates
  ajfs scan --hash --algo=sha1 --progress ~/database.ajfs /media/backups

  • Resume calculating file signature hashes.

    1

      ajfs resume --progress ~/database.ajfs

  • Update the snapshot to reflect the current file system hierarchy.

    1

      ajfs update --progress ~/database.ajfs

  • List a snapshot.

    1
2
3

      ajfs list mydata.ajfs

  ajfs tree mydata.ajfs

  • Search for matching entries.

    1
2
3
4
5
6
7
8

      # list all PDF files that have `go` in the filename
  ajfs search ~/mybooks.ajfs --type f --iname '*go*.pdf'

  # list all entries that match the regular expression
  ajfs search -i "\.txt$"

  # list all files smaller than 1GB
  ajfs search --type f --size -1G

  • See what has changed.

    1
2
3
4
5

      # diff the default ./db.ajfs and the root path it was created from
  ajfs diff snap1.ajfs

  # diff two snapshots
  ajfs diff snap1.ajfs snap2.ajfs

  • Find duplicates.

    1
2
3
4
5

      # find duplicate files
  ajfs dupes database.ajfs

  # find duplicate directory subtrees
  ajfs dupes --dirs database.ajfs

  • See what still needs to be backed up.

    1
2
3
4
5
6
7
8

      # what needs to be copied from my laptop to my nas
  ajfs tosync ~/laptop.ajfs ~/nas.ajfs

  # which files exist on the nas that has been deleted on my laptop
  ajfs tosync ~/nas.ajfs ~/laptop.ajfs

  # which files from my laptop has not yet been backed up on the nas regardless of filename or location
  ajfs tosync --hash ~/laptop.ajfs ~/nas.ajfs

  • Export the snapshot to other formats.

    1
2
3

      ajfs export database.ajfs export.csv

  ajfs export --format=json database.ajfs export.json

Disclaimer: This project is released under the MIT License. It is provided without warranty of any kind. By using this tool, you agree that the developer is not responsible for any issues, data loss, or system instability that may occur. Use it wisely and always backup your data first!

Projects
ajfs Go

Trending Tags

ajfs Go