rclone is an amazing tool to transfer data to/from all kind of storage elements. It is particularly handy when copying data from or to DESY Sync&Share (desycloud).
Configuring a nextcloud instance
An application password (token) is a extremely useful to access syncandshare without username/password. To generate and app-password visit for example https://syncandshare.desy.de/index.php/settings/user/security and scroll to the bottom of the page.
To configure access to syncandshare.desy.de (or your favorite alias) run rclone config:
rclone config
2021/04/08 09:20:17 NOTICE: Config file "/home/user/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> DULS
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[omitting most of the entries]
24 / SSH/SFTP Connection
\ "sftp"
25 / Webdav
\ "webdav"
26 / Yandex Disk
\ "yandex"
27 / http Connection
\ "http"
Storage> 25
** See help for webdav backend at: https://rclone.org/webdav/ **
URL of http host to connect to
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Connect to example.com
\ "https://example.com"
url> https://syncandshare.desy.de/remote.php/dav/files/<your-user-name>
Name of the Webdav site/service/software you are using
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Nextcloud
\ "nextcloud"
2 / Owncloud
\ "owncloud"
3 / Sharepoint
\ "sharepoint"
4 / Other site/service or software
\ "other"
vendor> 1
User name
Enter a string value. Press Enter for the default ("").
user> <your username or leave empty>
Password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> y
password: <your app password>
password: <your app password>
Bearer token instead of user/pass (eg a Macaroon)
Enter a string value. Press Enter for the default ("").
bearer_token> <leave empty>
Remote config
--------------------
[DULS]
type = webdav
url = https://syncandshare.desy.de/remote.php/dav/files/<your-user-name>
vendor = nextcloud
user = user
pass = ...
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
DULS webdav
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
This example created a remote named DULS (DESY UnLimited Space). The config file created can be found at ~/.config/rclone/rclone.conf:
[DULS] type = webdav url = https://syncandshare.desy.de/remote.php/dav/files/<username> vendor = nextcloud user = <username> pass = <app password> # <username> and <app password> contain the values specified with rclone config. Instead of running rclone config, you can also create the config file manually.
Copying files to syncandshare
To copy a single file to syncandshare (see https://rclone.org/commands/rclone_copy/):
rclone copy ipympl.ipynb DULS:Notebooks/ # NOT what you want: rclone copy ipympl.ipynb DULS:Notebooks/test.ipynb will create a FOLDER test.ipynb and copy ipympl.ipynb into that folder.
rclone will create folder on the fly if they don't exist. In this case, rclone created https://syncandshare.desy.de/index.php/apps/files/?dir=/Notebooks and uploaded the sample notebook file.
It's also possible to upload entire directories. For example:
rclone copy Notebooks DULS:Notebooks/
That will copy all files contained in Notebooks/ to the syncandshare folder Notebooks/.
Sync'ing folder to syncandshare
Before starting to sync folder please consult the documentation
rclone sync syncs the source to the destination, changing the destination only. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary (except duplicate objects, see below).
Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.
# test what rclone sync would do: rclone sync --dry-run Notebooks/ DULS:TestFolder/ 2021/04/08 09:56:03 NOTICE: 1_pure_performance.ipynb: Not copying as --dry-run [...] # create a clone of folder Notebooks on syncandshare: rclone sync --progress Notebooks/ DULS:TestFolder/ # removing a file and sync will also remove the file from syncandshare! rm Notebooks/00test.ipynb rclone sync Notebooks/ DULS:TestFolder/ # deletes 00test.ipynb from syncandshare # sync'ing a different folder will remove all files from remote! mkdir Blubber echo test > Blubber/test.txt rclone sync Blubber/ DULS:TestFolder/ # DULS:TestFolder/ will now only contain test.txt, everything else is gone!
Mounting syncandshare locally
rclone mount allows to create a mount in userspace. For example
# create a mount point, make sure it's NOT world readable mkdir --mode=700 -p /tmp/$USER/syncandshare # create the mount (interactively). It will run until being interrupted (e.g. CTRL-c). rclone mount DULS:/ /tmp/$USER/syncandshare # create the mount (interactively). It will run in the background until logout rclone mount --daemon DULS:/ /tmp/$USER/syncandshare # switch to a different terminal and create a new file echo test-mount > /tmp/$USER/syncandshare/TestFolder/test-mount.txt # the file will be "automatically" uploaded to syncandshare
Note: do not use a shared filesystem like $HOME or /beegfs as a mount point! The mount is always local to the machine you are logged in.
Note: do not forget to terminate the mount by CTRL+C (interactive mount) or fusermount -u /tmp/$USER/syncandshare when running daemonized.