This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
iridium_cluster:data [2014/05/13 14:57] florido |
iridium_cluster:data [2016/06/28 13:34] (current) florido [Generic storage] |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ===== Moving data to and from the cluster ===== | + | ====== Moving data to and from the cluster ====== |
Please read the section [[iridium_cluster:basic_information#Common files organization]] before going through this section. | Please read the section [[iridium_cluster:basic_information#Common files organization]] before going through this section. | ||
- | ==== Rules of thumb ==== | + | ===== Rules of thumb ===== |
:!: **Please read this carefully.** :!: | :!: **Please read this carefully.** :!: | ||
Line 15: | Line 15: | ||
* Don't expect contents of the ''scratch'' folder to be always there. We still have no policy for that but we will have meetings in which we decide about it. | * Don't expect contents of the ''scratch'' folder to be always there. We still have no policy for that but we will have meetings in which we decide about it. | ||
- | ==== Data transfer solutions ==== | + | ===== Data transfer solutions ===== |
Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists) | Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists) | ||
These marked with 8-) are my favourite --- //[[:Florido Paganelli]] 2013/08/27 20:20// | These marked with 8-) are my favourite --- //[[:Florido Paganelli]] 2013/08/27 20:20// | ||
+ | |||
+ | ==== Generic storage ==== | ||
=== Solution 1: scp,sftp,lsftp === | === Solution 1: scp,sftp,lsftp === | ||
Line 35: | Line 37: | ||
//Example:// | //Example:// | ||
- | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' | + | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' shared folders |
<code bash> | <code bash> | ||
Line 58: | Line 60: | ||
* does **not work** with GRID storage | * does **not work** with GRID storage | ||
- | //Example:// | + | Syntax: |
+ | <code bash> | ||
+ | rsync -avz -e 'ssh -l <username>' --progress source destination | ||
+ | </code> | ||
- | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' | + | However, the progress indicator is not very good and most of the time slows down |
+ | the transfers in the purpose of writing to standard output. | ||
+ | Therefore I suggest you either **redirect the standard error and output**: | ||
<code bash> | <code bash> | ||
- | rsync -avz --progress ubuntu-12.04.2-desktop-amd64.iso n12.iridium:/nfs/software/pp/ | + | rsync -avz -e 'ssh -l <username>' --progress source destination &> rsyncoutput.log |
</code> | </code> | ||
+ | Or even better, use **rsync own log file** instead: | ||
+ | <code bash> | ||
+ | rsync -avz -e 'ssh -l <username>' --log-file=rsyncoutput.log source destination | ||
+ | </code> | ||
+ | |||
+ | check the contents of the logfile now and then to see the status: | ||
+ | <code bash> | ||
+ | tail rsyncoutput.log | ||
+ | </code> | ||
+ | |||
+ | |||
+ | |||
+ | //Examples:// | ||
+ | |||
+ | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''pptest-iridium'' shared folders | ||
+ | |||
+ | <code bash> | ||
+ | rsync -avz -e 'ssh -l pflorido' --progress ubuntu-12.04.2-desktop-amd64.iso pptest-iridium.lunarc.lu.se:/nfs/software/pp/ | ||
+ | </code> | ||
+ | |||
+ | Note on the trailing slashes **/**: | ||
+ | |||
+ | source **without** trailing slash on source **will create** //localdir// remotely: | ||
+ | <code bash> | ||
+ | rsync -avz -e 'ssh -l pflorido' --progress localdir pptest-iridium.iridium:/nfs/software/pp/ | ||
+ | </code> | ||
+ | |||
+ | source **with** trailing slash on source **will NOT create** //localdir// remotely but will **copy the contents** of //localdir// remotely | ||
+ | <code bash> | ||
+ | rsync -avz -e 'ssh -l pflorido' --progress localdir/ pptest-iridium.iridium:/nfs/software/pp/ | ||
+ | </code> | ||
+ | |||
+ | Trailing slash on destination doesn't have any effect. | ||
=== Solution 3: FileZilla === | === Solution 3: FileZilla === | ||
Line 80: | Line 120: | ||
More about it: https://filezilla-project.org/download.php?type=client | More about it: https://filezilla-project.org/download.php?type=client | ||
+ | |||
+ | ==== GRID storage ==== | ||
=== Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) === | === Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) === | ||
Line 85: | Line 127: | ||
* **Pros:** | * **Pros:** | ||
* works with GRID storage | * works with GRID storage | ||
+ | * similar to cp | ||
* **Cons:** | * **Cons:** | ||
* doesn't work with ATLAS datasets (yet ;-) ) | * doesn't work with ATLAS datasets (yet ;-) ) | ||
- | * uncommon command line interface | ||
See also http://www.hep.lu.se/grid/localgroupdisk.html for more information on how to use Lund local GRID storage. | See also http://www.hep.lu.se/grid/localgroupdisk.html for more information on how to use Lund local GRID storage. | ||
Line 98: | Line 140: | ||
- | === Solution 5: dq2 tools === | + | === Solution 5: Rucio or dq2 tools === |
* **Pros:** | * **Pros:** | ||
* works with GRID storage | * works with GRID storage | ||
* **Cons:** | * **Cons:** | ||
- | * works with ATLAS datasets | + | * works only with ATLAS datasets |
- | * uncommon command line interface (but some are used to it) | + | |
- | //Example:// | + | If you have and ATLAS dataset, the best is to transfer it to the local |
- | <code> | + | LUND Grid storage first, and then to the cluster directly if needed. |
+ | To do that you need to submit a DaTRi request | ||
- | </code> | + | This page contains all you need to know on how to use the local storage: |
+ | http://www.hep.lu.se/grid/localgroupdisk.html | ||
+ | |||
+ | To move the dataset from any ATLAS grid storage to Iridium, you are recommended to use Rucio, the successor of DQ2. Use the following: | ||
+ | |||
+ | To enable RUCIO tools, you'll need to: | ||
+ | - copy and configure you GRID certificate on Iridium. | ||
+ | - run ''setupATLAS'' | ||
+ | - run ''localSetupRucioClients'' | ||
+ | - login to the GRID using ''arcproxy -S atlas'' or ''voms-proxy-init'' as one would do on //lxplus.cern.ch//. | ||
+ | |||
+ | The RUCIO official documentation is here: http://rucio.cern.ch/cli_examples.html | ||
+ | |||
+ | If you still want to use dq2 tools, here's how: | ||
+ | |||
+ | To enable dq2 tools, you'll need to: | ||
+ | - copy and configure you GRID certificate on Iridium. | ||
+ | - run ''setupATLAS'' | ||
+ | - run ''localSetupDQ2Client'' | ||
+ | - login to the GRID using ''arcproxy'' or ''voms-proxy-init'' as one would do on //lxplus.cern.ch// . | ||
+ | Information about dq2 on CERN Twiki (only visible if you have a CERN account): | ||
+ | https://twiki.cern.ch/twiki/bin/view/AtlasComputing/DQ2ClientsHowTo | ||
---- | ---- |