pfwiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Moving data to and from the cluster ======

Please read the section [[iridium_cluster:basic_information#Common files organization]] before going through this section.

===== Rules of thumb =====

:!: **Please read this carefully.** :!:

When moving data to the shared folders, please follow these common sense rules:
  * Create folders for everything you want to share.
  * If the data has been produced by you, is nice to create a folder with your name and place everything in it.
  * If the data belongs to some specific experiment, dataset or the like, create a folder name that is consistent with that and that is easy for everybody to understand what that is about.
  * Don't overdo. Only copy data you/your colleagues need. This is a shared facility.
  * Don't remove other user's files unless you advice them and they're ok with it. This is a shared facility.
  * Don't expect contents of the ''scratch'' folder to be always there. We still have no policy for that but we will have meetings in which we decide about it.

===== Data transfer solutions =====

Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists)

These marked with 8-) are my favourite  --- //[[:Florido Paganelli]] 2013/08/27 20:20//

==== Generic storage ====

=== Solution 1: scp,sftp,lsftp ===

  * **Pros:** 
    * easy
    * only needs terminal
    * available almost everywhere
    * progress indicator
  * **Cons:** 
    * not reliable. If connection goes down one must restart the entire transfer.
    * does **not work** with GRID storage
    * slow

//Example://

Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' shared folders

<code bash>
  scp ubuntu-12.04.2-desktop-i386.iso n12.iridium:/nfs/shared/pp/
</code>

=== Solution 2: rsync  ===

8-)

  * **Pros:** 
    * Reliable. If connection goes down will resume from where it stopped.
    * Minimizes amount of transferred data by compressing it
    * only needs terminal
    * available on most GNU/Linux platforms
    * a bit faster
  * **Cons:**
    * Awkward command line
    * bad logs
    * poor progress indicator on many files
    * available on windows but needs special installation
    * does **not work** with GRID storage

Syntax:
<code bash>
  rsync -avz -e 'ssh -l <username>' --progress source destination
</code>


//Examples://

Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''pptest-iridium'' shared folders

<code bash>
  rsync -avz -e 'ssh -l pflorido' --progress ubuntu-12.04.2-desktop-amd64.iso pptest-iridium.lunarc.lu.se:/nfs/software/pp/
</code>

Note on the trailing slashes **/**:

source **without** trailing slash on source **will create** //localdir// remotely:
<code bash>
  rsync -avz -e 'ssh -l pflorido' --progress localdir pptest-iridium.iridium:/nfs/software/pp/
</code>

source **with** trailing slash on source **will NOT create** //localdir// remotely but will **copy the contents** of //localdir// remotely
<code bash>
  rsync -avz -e 'ssh -l pflorido' --progress localdir/ pptest-iridium.iridium:/nfs/software/pp/
</code>

Trailing slash on destination doesn't have any effect.

=== Solution 3: FileZilla ===

  * **Pros:**
    * Reliable. Tries to resume if connection went down.
    * Visual interface
    * Available for both GNU/Linux and windows
  * **Cons:**
    * Visual interface :D
    * good logs
    * progress bar ^_^ 
    * does **not work** with GRID storage

More about it: https://filezilla-project.org/download.php?type=client

==== GRID storage ====

=== Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) ===

  * **Pros:**
    * works with GRID storage
    * similar to cp
  * **Cons:**
    * doesn't work with ATLAS datasets (yet ;-) )

See also http://www.hep.lu.se/grid/localgroupdisk.html for more information on how to use Lund local GRID storage.

//Example://

To copy files to/from the storage, use the ''srm:'' protocol and ''arccp'' tool: 

<code bash>arccp srm://srm.swegrid.se/atlas/disk/atlaslocalgroupdisk/lund/data11_7TeV/NTUP_SUSY/f354_m765_p486/data11_7TeV.00178109.physics_JetTauEtmiss.merge.NTUP_SUSY.f354_m765_p486_tid292683_00/NTUP_SUSY.292683._000131.root.1 file:///tmp/NTUP_SUSY.292683._000131.root.1 </code>


=== Solution 5: dq2 tools ===

  * **Pros:**
    * works with GRID storage
  * **Cons:**
    * works with ATLAS datasets

//Example://
<code>

</code>


----
pfwiki

User Tools

Site Tools

Page Tools
