iridium_cluster:data
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
iridium_cluster:data [2014/05/13 15:00] florido |
iridium_cluster:data [2014/06/04 11:16] florido [GRID storage] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ===== Moving data to and from the cluster ===== | + | ====== Moving data to and from the cluster ====== |
| Please read the section [[iridium_cluster:basic_information#Common files organization]] before going through this section. | Please read the section [[iridium_cluster:basic_information#Common files organization]] before going through this section. | ||
| - | ==== Rules of thumb ==== | + | ===== Rules of thumb ===== |
| :!: **Please read this carefully.** :!: | :!: **Please read this carefully.** :!: | ||
| Line 15: | Line 15: | ||
| * Don't expect contents of the ''scratch'' folder to be always there. We still have no policy for that but we will have meetings in which we decide about it. | * Don't expect contents of the ''scratch'' folder to be always there. We still have no policy for that but we will have meetings in which we decide about it. | ||
| - | ==== Data transfer solutions ==== | + | ===== Data transfer solutions ===== |
| Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists) | Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists) | ||
| Line 21: | Line 21: | ||
| These marked with 8-) are my favourite --- //[[:Florido Paganelli]] 2013/08/27 20:20// | These marked with 8-) are my favourite --- //[[:Florido Paganelli]] 2013/08/27 20:20// | ||
| - | === Generic transfer === | + | ==== Generic storage ==== |
| - | == Solution 1: scp,sftp,lsftp == | + | === Solution 1: scp,sftp,lsftp === |
| * **Pros:** | * **Pros:** | ||
| Line 37: | Line 37: | ||
| //Example:// | //Example:// | ||
| - | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' | + | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' shared folders |
| <code bash> | <code bash> | ||
| Line 43: | Line 43: | ||
| </code> | </code> | ||
| - | == Solution 2: rsync == | + | === Solution 2: rsync === |
| 8-) | 8-) | ||
| Line 60: | Line 60: | ||
| * does **not work** with GRID storage | * does **not work** with GRID storage | ||
| - | //Example:// | + | Syntax: |
| + | <code bash> | ||
| + | rsync -avz -e 'ssh -l <username>' --progress source destination | ||
| + | </code> | ||
| + | |||
| + | |||
| + | //Examples:// | ||
| - | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' | + | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''pptest-iridium'' shared folders |
| <code bash> | <code bash> | ||
| - | rsync -avz --progress ubuntu-12.04.2-desktop-amd64.iso n12.iridium:/nfs/software/pp/ | + | rsync -avz -e 'ssh -l pflorido' --progress ubuntu-12.04.2-desktop-amd64.iso pptest-iridium.lunarc.lu.se:/nfs/software/pp/ |
| </code> | </code> | ||
| + | Note on the trailing slashes **/**: | ||
| - | == Solution 3: FileZilla == | + | source **without** trailing slash on source **will create** //localdir// remotely: |
| + | <code bash> | ||
| + | rsync -avz -e 'ssh -l pflorido' --progress localdir pptest-iridium.iridium:/nfs/software/pp/ | ||
| + | </code> | ||
| + | |||
| + | source **with** trailing slash on source **will NOT create** //localdir// remotely but will **copy the contents** of //localdir// remotely | ||
| + | <code bash> | ||
| + | rsync -avz -e 'ssh -l pflorido' --progress localdir/ pptest-iridium.iridium:/nfs/software/pp/ | ||
| + | </code> | ||
| + | |||
| + | Trailing slash on destination doesn't have any effect. | ||
| + | |||
| + | === Solution 3: FileZilla === | ||
| * **Pros:** | * **Pros:** | ||
| Line 85: | Line 104: | ||
| ==== GRID storage ==== | ==== GRID storage ==== | ||
| - | == Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) == | + | === Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) === |
| * **Pros:** | * **Pros:** | ||
| * works with GRID storage | * works with GRID storage | ||
| + | * similar to cp | ||
| * **Cons:** | * **Cons:** | ||
| * doesn't work with ATLAS datasets (yet ;-) ) | * doesn't work with ATLAS datasets (yet ;-) ) | ||
| - | * uncommon command line interface | ||
| See also http://www.hep.lu.se/grid/localgroupdisk.html for more information on how to use Lund local GRID storage. | See also http://www.hep.lu.se/grid/localgroupdisk.html for more information on how to use Lund local GRID storage. | ||
| Line 102: | Line 121: | ||
| - | == Solution 5: dq2 tools == | + | === Solution 5: dq2 tools === |
| * **Pros:** | * **Pros:** | ||
| Line 108: | Line 127: | ||
| * **Cons:** | * **Cons:** | ||
| * works with ATLAS datasets | * works with ATLAS datasets | ||
| - | * uncommon command line interface (but some are used to it) | ||
| - | //Example:// | + | To enable dq2 tools, you'll need to: |
| - | <code> | + | - copy and configure you GRID certificate on Iridium. |
| + | - run ''setupATLAS'' | ||
| + | - run ''localSetupDQ2Client'' | ||
| + | - login to the GRID using ''arcproxy'' or ''voms-proxy-init'' as one would do on //lxplus.cern.ch// . | ||
| - | </code> | + | Information about dq2 on CERN Twiki (only visible if you have a CERN account): |
| + | https://twiki.cern.ch/twiki/bin/view/AtlasComputing/DQ2ClientsHowTo | ||
| ---- | ---- | ||
iridium_cluster/data.txt · Last modified: 2016/06/28 13:34 by florido
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
