User Tools

Site Tools


iridium_cluster:data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
Next revision Both sides next revision
iridium_cluster:data [2014/03/21 19:29]
florido created
iridium_cluster:data [2014/06/04 11:16]
florido [GRID storage]
Line 1: Line 1:
-===== Moving data to and from the cluster =====+====== Moving data to and from the cluster ​======
  
 Please read the section [[iridium_cluster:​basic_information#​Common files organization]] before going through this section. Please read the section [[iridium_cluster:​basic_information#​Common files organization]] before going through this section.
  
-==== Rules of thumb ====+===== Rules of thumb =====
  
 :!: **Please read this carefully.** :!: :!: **Please read this carefully.** :!:
Line 15: Line 15:
   * Don't expect contents of the ''​scratch''​ folder to be always there. We still have no policy for that but we will have meetings in which we decide about it.   * Don't expect contents of the ''​scratch''​ folder to be always there. We still have no policy for that but we will have meetings in which we decide about it.
  
-==== Data transfer solutions ====+===== Data transfer solutions ​=====
  
 Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists) Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists)
  
 These marked with 8-) are my favourite ​ --- //​[[:​Florido Paganelli]] 2013/08/27 20:20// These marked with 8-) are my favourite ​ --- //​[[:​Florido Paganelli]] 2013/08/27 20:20//
 +
 +==== Generic storage ====
  
 === Solution 1: scp,​sftp,​lsftp === === Solution 1: scp,​sftp,​lsftp ===
Line 35: Line 37:
 //​Example://​ //​Example://​
  
-Moving ''​ubuntu-12.04.2-desktop-amd64.iso''​ from my local machine to ''​n12.iridium''​+Moving ''​ubuntu-12.04.2-desktop-amd64.iso''​ from my local machine to ''​n12.iridium'' ​shared folders
  
-<​code>​+<​code ​bash>
   scp ubuntu-12.04.2-desktop-i386.iso n12.iridium:/​nfs/​shared/​pp/​   scp ubuntu-12.04.2-desktop-i386.iso n12.iridium:/​nfs/​shared/​pp/​
 </​code>​ </​code>​
Line 58: Line 60:
     * does **not work** with GRID storage     * does **not work** with GRID storage
  
-//Example://+Syntax: 
 +<code bash> 
 +  rsync -avz -e 'ssh -l <​username>'​ --progress source destination 
 +</code> 
 + 
 + 
 +//Examples://
  
-Moving ''​ubuntu-12.04.2-desktop-amd64.iso''​ from my local machine to ''​n12.iridium''​+Moving ''​ubuntu-12.04.2-desktop-amd64.iso''​ from my local machine to ''​pptest-iridium'' ​shared folders
  
-<​code>​ +<​code ​bash
-  rsync -avz --progress ubuntu-12.04.2-desktop-amd64.iso ​n12.iridium:/​nfs/​software/​pp/​+  rsync -avz -e 'ssh -l pflorido' ​--progress ubuntu-12.04.2-desktop-amd64.iso ​pptest-iridium.lunarc.lu.se:/​nfs/​software/​pp/​
 </​code>​ </​code>​
  
 +Note on the trailing slashes **/**:
 +
 +source **without** trailing slash on source **will create** //​localdir//​ remotely:
 +<code bash>
 +  rsync -avz -e 'ssh -l pflorido'​ --progress localdir pptest-iridium.iridium:/​nfs/​software/​pp/​
 +</​code>​
 +
 +source **with** trailing slash on source **will NOT create** //​localdir//​ remotely but will **copy the contents** of //​localdir//​ remotely
 +<code bash>
 +  rsync -avz -e 'ssh -l pflorido'​ --progress localdir/ pptest-iridium.iridium:/​nfs/​software/​pp/​
 +</​code>​
 +
 +Trailing slash on destination doesn'​t have any effect.
  
 === Solution 3: FileZilla === === Solution 3: FileZilla ===
Line 80: Line 101:
  
 More about it: https://​filezilla-project.org/​download.php?​type=client More about it: https://​filezilla-project.org/​download.php?​type=client
 +
 +==== GRID storage ====
  
 === Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) === === Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) ===
Line 85: Line 108:
   * **Pros:**   * **Pros:**
     * works with GRID storage     * works with GRID storage
 +    * similar to cp
   * **Cons:**   * **Cons:**
     * doesn'​t work with ATLAS datasets (yet ;-) )     * doesn'​t work with ATLAS datasets (yet ;-) )
-    * uncommon command line interface+ 
 +See also http://​www.hep.lu.se/​grid/​localgroupdisk.html for more information on how to use Lund local GRID storage.
  
 //​Example://​ //​Example://​
-<​code>​ 
  
-</​code>​+To copy files to/from the storage, use the ''​srm:''​ protocol and ''​arccp''​ tool:  
 + 
 +<code bash>​arccp srm://​srm.swegrid.se/​atlas/​disk/​atlaslocalgroupdisk/​lund/​data11_7TeV/​NTUP_SUSY/​f354_m765_p486/​data11_7TeV.00178109.physics_JetTauEtmiss.merge.NTUP_SUSY.f354_m765_p486_tid292683_00/​NTUP_SUSY.292683._000131.root.1 file:///​tmp/​NTUP_SUSY.292683._000131.root.1 ​</​code>​
  
  
Line 101: Line 127:
   * **Cons:**   * **Cons:**
     * works with ATLAS datasets     * works with ATLAS datasets
-    * uncommon command line interface (but some are used to it) 
  
-//Example:// +To enable dq2 tools, you'll need to: 
-<​code>​+  - copy and configure you GRID certificate on Iridium. 
 +  - run ''​setupATLAS''​ 
 +  - run ''​localSetupDQ2Client''​ 
 +  - login to the GRID using ''​arcproxy''​ or ''​voms-proxy-init''​ as one would do on //lxplus.cern.ch// 
  
-</code>+Information about dq2 on CERN Twiki (only visible if you have a CERN account): 
 +https://​twiki.cern.ch/​twiki/​bin/​view/​AtlasComputing/​DQ2ClientsHowTo
  
  
 ---- ----
iridium_cluster/data.txt · Last modified: 2016/06/28 13:34 by florido

Accessibility Statement