Rclone: Difference between revisions

From David's Wiki
 
(7 intermediate revisions by the same user not shown)
Line 1: Line 1:
rclone<br>
rclone is a super useful tool for copying files to cloud storage such as Google Drive or S3-based object stores.
rsync for cloud storage<br>
[https://rclone.org/ https://rclone.org/]<br>
[https://rclone.org/ https://rclone.org/]<br>
Rclone is often faster than the built in file explorer due to handling multiple transfers (e.g. <code>--transfers 25</code>).


==Install==
==Install==
You can download a deb or rpm from their website.<br>
See https://rclone.org/install/<br>
Or you can run the following script
For Linux, you can download a deb or rpm from their website.<br>
<pre>
If you do not have sudo, you can download the binary and put it in your <code>$HOME/.local/bin</code> folder.
curl https://rclone.org/install.sh | sudo bash
</pre>
If you do not have sudo, you can download the binary and put it in your <code>~/bin</code> folder.


For Windows, rclone is also [https://chocolatey.org/packages/rclone available on Chocolatey]
For Windows, rclone is also [https://chocolatey.org/packages/rclone available on Chocolatey] and WinGET
<pre>
<pre>
choco install rclone winfsp -y
choco install rclone winfsp -y
</pre>
</pre>
* <code>winfsp</code> is needed for <code>rclone mount</code>
* <code>winfsp</code> is needed for <code>rclone mount</code>


Line 30: Line 27:
===Common Flags===
===Common Flags===
[https://rclone.org/flags/ flags]
[https://rclone.org/flags/ flags]
* <code>-n, --dry-run</code>
* <code>-P, --progress</code> show progress (transferred, ETA, elapsed)
* <code>-P, --progress</code> show progress (transferred, ETA, elapsed)
* <code>-n, --dry-run</code>
* <code>--stats-one-line</code> show only the total stats
* <code>--transfers N</code> do N simultaneous file transfers (defaults to 4)
* <code>--checkers N</code> (defaults to 8)


===Mount===
===Mount===
Line 47: Line 47:
** Just append <code>--vfs-cache-mode full</code> to the end of your mount.
** Just append <code>--vfs-cache-mode full</code> to the end of your mount.
**: The default cache is at <code>~/.cache/rclone/</code>.  
**: The default cache is at <code>~/.cache/rclone/</code>.  
** See [https://rclone.org/commands/rclone_mount/#file-caching rclone mount file caching] for more options.
** See [https://rclone.org/commands/rclone_mount/#vfs-file-caching vfs file caching] for more options.
* If you have unlimited data or are using a local mount, you may want to reduce the directory index cache time.
* If you have unlimited data or are using a local mount, you may want to reduce the directory index cache time.
** The default is <code>--dir-cache-time 5m0s</code> for 5 minutes.
** The default is <code>--dir-cache-time 5m0s</code> for 5 minutes.
Line 112: Line 112:
Then run:
Then run:
<pre>
<pre>
mkdir ${HOME}/terpmail
systemctl --user daemon-reload
systemctl --user daemon-reload
systemctl --user enable terpmail
systemctl --user enable terpmail

Latest revision as of 17:30, 31 August 2023

rclone is a super useful tool for copying files to cloud storage such as Google Drive or S3-based object stores. https://rclone.org/

Rclone is often faster than the built in file explorer due to handling multiple transfers (e.g. --transfers 25).

Install

See https://rclone.org/install/
For Linux, you can download a deb or rpm from their website.
If you do not have sudo, you can download the binary and put it in your $HOME/.local/bin folder.

For Windows, rclone is also available on Chocolatey and WinGET

choco install rclone winfsp -y
  • winfsp is needed for rclone mount

Usage

Setup Backends

rclone config
Note
  • Backends and their configuration options are stored in your config file.
    • $HOME/.config/rclone/rclone.conf

Common Flags

flags

  • -n, --dry-run
  • -P, --progress show progress (transferred, ETA, elapsed)
  • --stats-one-line show only the total stats
  • --transfers N do N simultaneous file transfers (defaults to 4)
  • --checkers N (defaults to 8)

Mount

rclone mount documentation

mkdir /path/to/local/mount
rclone mount remote:path/to/files /path/to/local/mount [flags]
Unmount
  • fusermount -u /path/to/local/mount
Notes
  • I strongly recommend using some kind of cache by setting the vfs cache mode or using a cache backend.
    • Just append --vfs-cache-mode full to the end of your mount.
      The default cache is at ~/.cache/rclone/.
    • See vfs file caching for more options.
  • If you have unlimited data or are using a local mount, you may want to reduce the directory index cache time.
    • The default is --dir-cache-time 5m0s for 5 minutes.
      I usually set 15 seconds for local SFTP mounts.
Example

Windows:

rclone mount t440s-server-local: H: --vfs-cache-mode full --dir-cache-time 5m0s

Sync

rclone sync documentation

Make source and dest identical, modifying destination only.
Note: This is like copy but it will delete files in the destination.

rclone sync source:path dest:path [flags]
Flags
      --create-empty-src-dirs   Create empty source dirs on destination after sync
  -h, --help                    help for sync

Copy

rclone copy

Copy files from source to dest, skipping already copied

rclone copy source:path dest:path [flags]
Notes
  • This will overwrite existing files which are different
    • Use --ignore-existing to skip existing files
    • Use --backup-dir <path> to move existing files which would be modified to another folder

Crypt

https://rclone.org/crypt/
Crypt allows you to encrypt the files you backup in the cloud. This is similar to Cryptomator but you don't need to store the encrypted files locally.

Crypt creates fake remotes (Crypt remotes) which represent decrypted versions of paths in real remotes (e.g. Google Drive).

Mount on Startup

You can mount on startup using a systemd service:

# ~/.config/systemd/user/terpmail.service
[Unit]
Description=rclone terpmail

[Service]
Type=simple
Restart=always
ExecStart=/usr/bin/rclone mount terpmail: ${HOME}/terpmail --vfs-cache-mode full
ExecStop=/bin/fusermount -u ${HOME}/terpmail

[Install]
WantedBy=default.target

Then run:

mkdir ${HOME}/terpmail
systemctl --user daemon-reload
systemctl --user enable terpmail
systemctl --user start terpmail

On Windows, you can create a service using nssm:

  1. Install nssm
    choco install nssm -y
  2. Create a service
    nssm install my_service
Notes
  • If using nssm, you will need to specify the full path to the config since it runs as another user.