User Tools

Site Tools


iridium_cluster:data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
iridium_cluster:data [2014/05/13 15:01]
florido
iridium_cluster:data [2015/04/24 14:38]
florido [GRID storage]
Line 21: Line 21:
 These marked with 8-) are my favourite ​ --- //​[[:​Florido Paganelli]] 2013/08/27 20:20// These marked with 8-) are my favourite ​ --- //​[[:​Florido Paganelli]] 2013/08/27 20:20//
  
-==== Generic ​transfer ​====+==== Generic ​storage ​====
  
 === Solution 1: scp,​sftp,​lsftp === === Solution 1: scp,​sftp,​lsftp ===
Line 37: Line 37:
 //​Example://​ //​Example://​
  
-Moving ''​ubuntu-12.04.2-desktop-amd64.iso''​ from my local machine to ''​n12.iridium''​+Moving ''​ubuntu-12.04.2-desktop-amd64.iso''​ from my local machine to ''​n12.iridium'' ​shared folders
  
 <code bash> <code bash>
Line 60: Line 60:
     * does **not work** with GRID storage     * does **not work** with GRID storage
  
-//Example://+Syntax: 
 +<code bash> 
 +  rsync -avz -e 'ssh -l <​username>'​ --progress source destination 
 +</code> 
 + 
 + 
 +//Examples://
  
-Moving ''​ubuntu-12.04.2-desktop-amd64.iso''​ from my local machine to ''​n12.iridium''​+Moving ''​ubuntu-12.04.2-desktop-amd64.iso''​ from my local machine to ''​pptest-iridium'' ​shared folders
  
 <code bash> <code bash>
-  rsync -avz --progress ubuntu-12.04.2-desktop-amd64.iso ​n12.iridium:/​nfs/​software/​pp/​+  rsync -avz -e 'ssh -l pflorido' ​--progress ubuntu-12.04.2-desktop-amd64.iso ​pptest-iridium.lunarc.lu.se:/​nfs/​software/​pp/​
 </​code>​ </​code>​
  
 +Note on the trailing slashes **/**:
 +
 +source **without** trailing slash on source **will create** //​localdir//​ remotely:
 +<code bash>
 +  rsync -avz -e 'ssh -l pflorido'​ --progress localdir pptest-iridium.iridium:/​nfs/​software/​pp/​
 +</​code>​
 +
 +source **with** trailing slash on source **will NOT create** //​localdir//​ remotely but will **copy the contents** of //​localdir//​ remotely
 +<code bash>
 +  rsync -avz -e 'ssh -l pflorido'​ --progress localdir/ pptest-iridium.iridium:/​nfs/​software/​pp/​
 +</​code>​
 +
 +Trailing slash on destination doesn'​t have any effect.
  
 === Solution 3: FileZilla === === Solution 3: FileZilla ===
Line 83: Line 102:
 More about it: https://​filezilla-project.org/​download.php?​type=client More about it: https://​filezilla-project.org/​download.php?​type=client
  
-===== GRID storage ​=====+==== GRID storage ====
  
 === Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) === === Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) ===
Line 89: Line 108:
   * **Pros:**   * **Pros:**
     * works with GRID storage     * works with GRID storage
 +    * similar to cp
   * **Cons:**   * **Cons:**
     * doesn'​t work with ATLAS datasets (yet ;-) )     * doesn'​t work with ATLAS datasets (yet ;-) )
-    * uncommon command line interface 
  
 See also http://​www.hep.lu.se/​grid/​localgroupdisk.html for more information on how to use Lund local GRID storage. See also http://​www.hep.lu.se/​grid/​localgroupdisk.html for more information on how to use Lund local GRID storage.
Line 102: Line 121:
  
  
-=== Solution 5: dq2 tools ===+=== Solution 5: Rucio or dq2 tools ===
  
   * **Pros:**   * **Pros:**
     * works with GRID storage     * works with GRID storage
   * **Cons:**   * **Cons:**
-    * works with ATLAS datasets +    * works only with ATLAS datasets
-    * uncommon command line interface (but some are used to it)+
  
-//​Example://​ +If you have and ATLAS dataset, the best is to transfer it to the local 
-<​code>​+LUND Grid storage first, and then to the cluster directly if needed.  
 +To do that you need to submit a DaTRi request
  
-</code>+This page contains all you need to know on how to use the local storage:  
 +http://​www.hep.lu.se/​grid/​localgroupdisk.html 
 + 
 +To move the dataset from any ATLAS grid storage to Iridium, you are recommended to use Rucio, the successor of DQ2. Use the following:​ 
 + 
 +To enable RUCIO tools, you'll need to: 
 +  - copy and configure you GRID certificate on Iridium. 
 +  - run ''​setupATLAS''​ 
 +  - run ''​localSetupRucioClients''​ 
 +  - login to the GRID using ''​arcproxy -S atlas''​ or ''​voms-proxy-init''​ as one would do on //​lxplus.cern.ch//​.  
 + 
 +The RUCIO official documentation is here: http://​rucio.cern.ch/​cli_examples.html 
 + 
 +If you still want to use dq2 tools, here's how: 
 + 
 +To enable dq2 tools, you'll need to: 
 +  - copy and configure you GRID certificate on Iridium. 
 +  - run ''​setupATLAS''​ 
 +  - run ''​localSetupDQ2Client''​ 
 +  - login to the GRID using ''​arcproxy''​ or ''​voms-proxy-init''​ as one would do on //​lxplus.cern.ch//​ . 
  
 +Information about dq2 on CERN Twiki (only visible if you have a CERN account):
 +https://​twiki.cern.ch/​twiki/​bin/​view/​AtlasComputing/​DQ2ClientsHowTo
  
 ---- ----
iridium_cluster/data.txt · Last modified: 2016/06/28 13:34 by florido

Accessibility Statement