Rclone

From David's Wiki
Revision as of 03:19, 22 June 2020 by David (talk | contribs) (→‎Mount)
\( \newcommand{\P}[]{\unicode{xB6}} \newcommand{\AA}[]{\unicode{x212B}} \newcommand{\empty}[]{\emptyset} \newcommand{\O}[]{\emptyset} \newcommand{\Alpha}[]{Α} \newcommand{\Beta}[]{Β} \newcommand{\Epsilon}[]{Ε} \newcommand{\Iota}[]{Ι} \newcommand{\Kappa}[]{Κ} \newcommand{\Rho}[]{Ρ} \newcommand{\Tau}[]{Τ} \newcommand{\Zeta}[]{Ζ} \newcommand{\Mu}[]{\unicode{x039C}} \newcommand{\Chi}[]{Χ} \newcommand{\Eta}[]{\unicode{x0397}} \newcommand{\Nu}[]{\unicode{x039D}} \newcommand{\Omicron}[]{\unicode{x039F}} \DeclareMathOperator{\sgn}{sgn} \def\oiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x222F}\,}{\unicode{x222F}}{\unicode{x222F}}{\unicode{x222F}}}\,}\nolimits} \def\oiiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x2230}\,}{\unicode{x2230}}{\unicode{x2230}}{\unicode{x2230}}}\,}\nolimits} \)

rclone
rsync for cloud storage
https://rclone.org/

Install

You can download a deb or rpm from their website.
Or you can run the following script

curl https://rclone.org/install.sh | sudo bash

If you do not have sudo, you can download the binary and put it in your ~/bin folder.

For Windows, rclone is also available on Chocolatey

choco install rclone winfsp -y
  • winfsp is needed for rclone mount

Usage

Setup Backends

rclone config
Note
  • Backends and their configuration options are stored in your config file.
    • $HOME/.config/rclone/rclone.conf

Common Flags

flags

  • -P, --progress show progress (transferred, ETA, elapsed)
  • -n, --dry-run

Mount

rclone mount documentation

mkdir /path/to/local/mount
rclone mount remote:path/to/files /path/to/local/mount [flags]
Unmount
  • fusermount -u /path/to/local/mount
Notes
  • I strongly recommend using some kind of cache by setting the vfs cache mode or using a cache backend.
    • Just append --vfs-cache-mode full to the end of your mount.
      The default cache is at ~/.cache/rclone/.
    • See rclone mount file caching for more options.
  • If you have unlimited data or are using a local mount, you may want to reduce the directory index cache time.
    • The default is --dir-cache-time 5m0s for 5 minutes.
      I usually set 15 seconds for local SFTP mounts.
Example

Windows:

rclone mount t440s-server-local: H: --vfs-cache-mode full --dir-cache-time 5m0s

Sync

rclone sync documentation

Make source and dest identical, modifying destination only.
Note: This is like copy but it will delete files in the destination.

rclone sync source:path dest:path [flags]
Flags
      --create-empty-src-dirs   Create empty source dirs on destination after sync
  -h, --help                    help for sync

Copy

rclone copy

Copy files from source to dest, skipping already copied

rclone copy source:path dest:path [flags]
Notes
  • This will overwrite existing files which are different
    • Use --ignore-existing to skip existing files
    • Use --backup-dir <path> to move existing files which would be modified to another folder

Crypt

https://rclone.org/crypt/
Crypt allows you to encrypt the files you backup in the cloud. This is similar to Cryptomator but you don't need to store the encrypted files locally.

Crypt creates fake remotes (Crypt remotes) which represent decrypted versions of paths in real remotes (e.g. Google Drive).

Mount on Startup

You can mount on startup using a systemd service:

# /etc/systemd/system/terpmail.service
[Unit]
Description=rclone terpmail
After=network.target

[Service]
Type=simple
Restart=always
User=david
Group=david
ExecStart=/usr/bin/rclone mount terpmail: ${HOME}/terpmail --vfs-cache-mode full
ExecStop=/bin/fusermount -u ${HOME}/terpmail

[Install]
WantedBy=multi-user.target

On Windows, you can create a service using nssm:

  1. Install nssm
    choco install nssm -y
  2. Create a service
    nssm install my_service
Notes
  • If using nssm, you will need to specify the full path to the config since it runs as another user.