iridium_cluster:data
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
iridium_cluster:data [2014/06/04 11:16] florido [GRID storage] |
iridium_cluster:data [2016/06/28 13:34] florido [Generic storage] |
||
|---|---|---|---|
| Line 64: | Line 64: | ||
| rsync -avz -e 'ssh -l <username>' --progress source destination | rsync -avz -e 'ssh -l <username>' --progress source destination | ||
| </code> | </code> | ||
| + | |||
| + | However, the progress indicator is not very good and most of the time slows down | ||
| + | the transfers in the purpose of writing to standard output. | ||
| + | Therefore I suggest you either **redirect the standard error and output**: | ||
| + | |||
| + | <code bash> | ||
| + | rsync -avz -e 'ssh -l <username>' --progress source destination &> rsyncoutput.log | ||
| + | </code> | ||
| + | |||
| + | Or even better, use **rsync own log file** instead: | ||
| + | <code bash> | ||
| + | rsync -avz -e 'ssh -l <username>' --log-file=rsyncoutput.log source destination | ||
| + | </code> | ||
| + | |||
| + | check the contents of the logfile now and then to see the status: | ||
| + | <code bash> | ||
| + | tail rsyncoutput.log | ||
| + | </code> | ||
| + | |||
| Line 121: | Line 140: | ||
| - | === Solution 5: dq2 tools === | + | === Solution 5: Rucio or dq2 tools === |
| * **Pros:** | * **Pros:** | ||
| * works with GRID storage | * works with GRID storage | ||
| * **Cons:** | * **Cons:** | ||
| - | * works with ATLAS datasets | + | * works only with ATLAS datasets |
| + | |||
| + | If you have and ATLAS dataset, the best is to transfer it to the local | ||
| + | LUND Grid storage first, and then to the cluster directly if needed. | ||
| + | To do that you need to submit a DaTRi request | ||
| + | |||
| + | This page contains all you need to know on how to use the local storage: | ||
| + | http://www.hep.lu.se/grid/localgroupdisk.html | ||
| + | |||
| + | To move the dataset from any ATLAS grid storage to Iridium, you are recommended to use Rucio, the successor of DQ2. Use the following: | ||
| + | |||
| + | To enable RUCIO tools, you'll need to: | ||
| + | - copy and configure you GRID certificate on Iridium. | ||
| + | - run ''setupATLAS'' | ||
| + | - run ''localSetupRucioClients'' | ||
| + | - login to the GRID using ''arcproxy -S atlas'' or ''voms-proxy-init'' as one would do on //lxplus.cern.ch//. | ||
| + | |||
| + | The RUCIO official documentation is here: http://rucio.cern.ch/cli_examples.html | ||
| + | |||
| + | If you still want to use dq2 tools, here's how: | ||
| To enable dq2 tools, you'll need to: | To enable dq2 tools, you'll need to: | ||
| Line 134: | Line 172: | ||
| - login to the GRID using ''arcproxy'' or ''voms-proxy-init'' as one would do on //lxplus.cern.ch// . | - login to the GRID using ''arcproxy'' or ''voms-proxy-init'' as one would do on //lxplus.cern.ch// . | ||
| - | Information about dq2 on CERN Twiki: | + | Information about dq2 on CERN Twiki (only visible if you have a CERN account): |
| https://twiki.cern.ch/twiki/bin/view/AtlasComputing/DQ2ClientsHowTo | https://twiki.cern.ch/twiki/bin/view/AtlasComputing/DQ2ClientsHowTo | ||
| - | |||
| ---- | ---- | ||
iridium_cluster/data.txt · Last modified: 2016/06/28 13:34 by florido
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
