Welcome to Hoard!

hoard is a program for backing up files from across a filesystem into a single directory and restoring them later.

Most people will know these programs as "dotfile managers," where dotfiles are configuration files on *nix (read: non-Windows) systems. Files on *nix systems are marked as hidden by starting the file name with a dot (.).

hoard aims to be a little more useful than other dotfile managers:

  1. Many dotfile managers store files in a structure based on their path relative to the user's home directory. This is useful in most cases, but can cause problems when wanted to share files across systems that don't use the same paths, e.g., Windows and Linux. hoard instead namespaces files based on the "Hoard" and "Pile" they are configured in, then relative to the root of the Pile. This makes it easy to backup and restore files to very different locations.

  2. Most dotfile managers do not prevent you from accidentally destructive behavior. See Checks for more information.

Environment

Not to be confused with an environment variable. An Environment is an identifiable system configuration consisting of zero or more each of the following:

  • Operating system
  • Hostname
  • Environment variable
  • Executables in $PATH
  • Existing paths (folders/files) on the system

Multiple Environments can be mixed and matched in Environment Strings when defining what paths to use for a given Pile. Some Environments may be mutually exclusive with certain others.

Pile

A single file or directory with multiple possible places where it can be found, depending on the system configuration. The path to use is determined by the best matching Environment String.

Hoard

A collection of one or more Piles that form a logical unit.

Examples

Consider this configuration snippet (see Configuration File for more explanation):

exclusivity = [
    ["neovim", "vim"],
]

[envs]
[envs.neovim]
    exe_exists = ["nvim", "nvim-qt"]
[envs.unix]
    os = ["linux", "freebsd"]
    env = [
        { var = "HOME" },
        { var = "XDG_CONFIG_HOME" }
    ]
[envs.vim]
    # Detect "vim" if AT LEAST one of `vim` or `gvim` exists in $PATH.
    exe_exists = ["vim", "gvim"]
[envs.windows]
    os = ["windows"]

[hoards]
[hoards.vim]
    [hoards.vim.init]
        "unix|neovim" = "${XDG_CONFIG_HOME}/nvim/init.vim"
        "unix|vim" = "${HOME}/.vimrc"
        "windows|neovim" = "${LOCALAPPDATA}\\nvim\\init.vim"
        "windows|vim" = "${USERPROFILE}\\.vim\\_vimrc"
    [hoards.vim.configdir]
        "windows|neovim" = "${LOCALAPPDATA}\\nvim\\config"
        "windows|vim" = "${USERPROFILE}\\.vim\\config"
        "unix|neovim" = "${XDG_CONFIG_HOME}/nvim/config"
        "unix|vim" = "${HOME}/.vim/config"
  • Environments: neovim, unix, vim, windows; neovim and vim are mutually exclusive.
  • Hoards: just one, called vim, containing two named Piles.
  • Piles: init and configdir; init is the entry config file for a Vim program, while configdir is a directory containing more config files loaded by init.

Take a closer look at the init Pile. There are four possible paths the file can be at, based on a combination of which operating system it is running on and whether Neovim or Vim is installed. The exclusivity line tells Hoard to prefer Neovim if both are present.

File Locations

This page explains what files are generated by hoard and where they can be found.

In general, hoard uses the directories library, using the config_dir and the data_dir of the ProjectDirs struct.

Config Directory

The configuration directory holds the configuration file (config.toml) as well as other local-only configuration data.

  • Linux/BSD: $XDG_CONFIG_HOME/hoard/ or $HOME/.config/hoard/
  • macos: $HOME/Library/Application Support/com.shadow53.hoard/
  • Windows: C:\Users\$USER\AppData\Roaming\shadow53\hoard\

Config File

The configuration file (config.toml) contains the environment and hoard definitions, along with all related configuration. Follow the link in the previous sentence for more about the configuration file format.

UUID File

The UUID file (uuid) contains a unique identifier for the current system. This is used when performing pre-operation checks. Files relating to this UUID are synchronized between machines using whatever synchronization mechanism you use to synchronize hoards between machines, but nowhere else. This UUID is not used to identify your machines to any service, only the hoard program.

Hoard Data Directory

The hoard data directory contains all backed up hoard files, along with other files that should be synchronized with the hoard files.

  • Linux/BSD: $XDG_DATA_HOME/hoard/ or /home/$USER/.local/share/hoard/
  • macos: $HOME/Library/Application Support/com.shadow53.hoard/
  • Windows: C:\Users\$USER\AppData\Roaming\shadow53\hoard\data\

Hoard Files

All files backed up by hoard are stored in the data directory, in a subdirectory called hoards.

The files are organized according to the names of the hoard and pile they are configured under.

As an example, consider the following real configuration:

[hoards.custom_fonts]
    "unix" = "/home/shadow53/.local/share/fonts"

[hoards.fish]
    [hoards.fish.confdir]
        "unix" = "/home/shadow53/.config/fish/conf.d"
    [hoards.fish.functions]
        "unix" = "/home/shadow53/.config/fish/functions"

[hoards.newsboat]
    "unix" = "/home/shadow53/.newsboat/config"

[hoards.qemu]
    [hoards.qemu.script]
        "unix" = "/home/shadow53/.bin/vm"
    [hoards.qemu.configs]
        "unix" = "/home/shadow53/.config/qemu"

These hoards/piles are stored in the following locations:

$data_dir
├─ custom_fonts/
├─ fish
│   ├─ confdir/
│   └─ functions/
├─ newsboat/
└─ qemu/
    ├─ script
    └─ configs/

History Files

There are currently two types of history-related files stored by hoard, both of which are used in pre-operation consistency checks. All history-related files are stored in a subdirectory history/{uuid} in the data directory, where uuid is the generated uuid of the current system.

  • Last Paths: a single file last_paths.json.
  • Operations: date-stamped JSON files with details of which files were modified during a given operation and what the checksum was for each file.

Command-Line Tool

This section describes the usage and behavior of the command-line tool hoard.

Installation

This page lists the supported methods of installing hoard.

Cargo

If you have cargo and the Rust toolchain installed, you can install hoard with the following command:

cargo install hoard

Flags

Flags can be used with any subcommand and must be specified before any subcommand.

  • --help: View the program's help message.
  • -V/--version: Print the installed version of hoard.
  • -c/--config-file: Path to (non-default) configuration file.
  • -h/--hoards-root: Path to (non-default) hoards root directory.

Subcommands

  • Backup: hoard [flags...] backup [name] [name] [...]
    • Back up the specified hoard(s). If no name is specified, all hoards are backed up.
  • Restore: hoard [flags...] restore [name] [name] [...]
    • Restore the specified hoard(s). If no name is specified, all hoards are restored.
  • Validate: hoard [flags...] validate
    • Attempt to parse the default configuration file (or the one provided via --config-file) Exits with code 0 if the config is valid.
  • Cleanup: hoard [flags...] cleanup

Logging

Output verbosity is controlled by the logging level. You can set the logging level with the HOARD_LOG environment variable. Valid values (in decreasing verbosity) are:

  • trace
  • debug
  • info
  • warn
  • error

The default logging level is info for release builds and debug for debugging builds.

Pre-Operation Checks

To help protect against accidentally overwriting or deleting files, hoard runs some consistency checks prior to running any operations.

To skip running the checks, run hoard with the --force flag. There is not currently a way to disable individual checks.

Last Paths

This check compares the paths used previously with a given hoard to the ones resolved for the current operation. If any of these paths differ, a warning is displayed and the operation(s) canceled.

Remote Operations

By default, hoard logs information about successful operations to a directory that is intended to be synchronized with the main hoards directory. This information is used to determine if a given file was last modified by a remote system. If so, a warning is displayed and the operation(s) canceled.

Configuration File

This section describes the configuration file format, with examples.

Environments

The path used for a given pile depends on the best matching environment(s) for a configured path. This page discusses how to define environments. For how to use them with hoards/piles, see Hoards and Piles.

Environments can be matched on one or more of five possible factors:

  • os: Operating System
  • env: Environment variables
    • Can match on just existence or also a specific value.
  • hostname: The system hostname.
  • exe_exists: Whether an executable file exists in $PATH.
  • path_exists: Whether something exists (one of) the given path(s).

All the above factors can be written using two-dimensional array syntax. That is, ["foo", ["bar, "baz"]] is interpreted as (foo) OR (bar AND baz), in whatever way applies to that given factor.

It is an error to include an AND condition for os or hostname, as a system can only have one of each.

[envs]
[envs.example_env]
    # Matching something *nix-y
    os = ["linux", "freebsd"]
    # Either sed and sh, or bash, must exist
    exe_exists = ["bash", ["sh", "sed"]]
    # Require both $HOME to exist and $HOARD_EXAMPLE_ENV to equal YES.
    # Note the double square brackets that indicate AND instead of OR.
    env = [[
      { var = "HOME" },
      { var = "HOARD_EXAMPLE_ENV", expected = "YES" },
    ]]

Exclusivity

The exclusivity lists indicate names of environments that are considered mutually exclusive to each other -- that is, cannot appear in the same environment condition -- and the order indicates which one(s) have precedence when matching environments.

See the example config file for a more thorough example.

exclusivity = [
    # Assuming all else the same, an environment condition string with "neovim" will take
    # precedence over one with "vim", which takes precedence over one with "emacs".
    ["neovim", "vim", "emacs"]
]

Hoards and Piles

Hoards consist of one or more piles, where each pile is a mapping of environment condition strings to paths on the filesystem.

An environment condition string is one or more environment names separated by pipes. The system must match ALL environments in the string in order for the associated path to be considered.

The following rules determine which path to use for a pile:

  1. The condition string with the most environments wins.
  2. If multiple conditions tie for most environments, the exclusivity list is used to determine if one takes precedence.
  3. If multiple conditions have the same precedence, an error is printed and hoard exits.
  4. If no conditions match, the pile is skipped and a warning is printed.
[hoards]
# This hoard consists of a single anonymous pile
[hoards.simple_hoard]
    # This is "foo" and "bar" separated by a pipe character (`|`).
    # It will use this path if the system matches both environments "foo" and "bar".
    "foo|bar" = "/path/to/a/thing"
    # This path is considered if the system matches the environment "baz".
    # It will use this path if one of "foo" or "bar" doesn't match. Otherwise, "foo|bar"
    # takes precedence because it is a longer condition (more environments to match).
    "baz" = "/some/different/path"

[hoards.complex_hoard]
# This hoard consists of two named piles: "first" and "second".
[hoards.complex_hoard.first]
    "foo|bar" = "/some/path/first"
    "baz" = "/some/different/path/first"
[hoards.complex_hoard.second]
    "foo|bar" = "/some/path/second"
    "baz" = "/some/different/path/second"

Environment Variables

Paths may contain environment variables. Environment variables must be written as ${ENVVAR}, where ENVVAR is the environment variable. As an example, the following hoard could be used to back up a user's Documents folder on Linux and Windows.

[hoards.documents]
    "linux" = "${HOME}/Documents"
    "windows" = "${USERPROFILE}/Documents"

For the user myuser, this expands to the following:

[hoards.documents]
    "linux" = "/home/myuser/Documents"
    "windows" = "C:/Users/myuser/Documents"

If the environment variable does not exist (i.e. is not defined), an error is returned and the operation is canceled.

Limitations

  1. There is no support for default values, i.e. ${MYVAR:-"/some/default"}

Pile Configuration

Pile configuration can be defined at three different levels:

  1. Globally
  2. Per-Hoard
  3. Per-Pile

For a given Pile, any/all three of the levels of configuration are "layered" together, as appropriate for each configuration item:

  • Ignore patterns are merged and deduplicated.
  • Encryption settings will use the most-specific settings.

Ignore Patterns

Set ignore to a list of glob patterns indicating files and folders to ignore. These lists will be merged across all levels of configuration.

# ... snip env definitions of "foo" and "bar" ...

# Top-level config, applies to all hoards
[config]
    # Ignore the .git folder at any depth
    ignore = ["**/.git"]

[hoards]
[hoards.anon_hoard]
    "foo" = "/some/path"
    "bar" = "/some/other/path"
[hoards.anon_hoard.config]
    ignore = [
        "**/.*", # Ignore all hidden files on Linux/macOS
        "*.log", # Ignore all top-level log files
    ]
[hoards.named_hoard]
[hoards.named_hoard.config]
    ignore = ["ignore-in-named-only"]
[hoards.named_hoard.pile1]
    "foo" = "/some/named/path"
    "bar" = "/another/named/path"