User Tools

Site Tools


aurora_cluster:moving_data:grid

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
aurora_cluster:moving_data:grid [2017/07/31 17:44]
florido
aurora_cluster:moving_data:grid [2017/10/05 10:50] (current)
florido [Downloading data from SE-SNIC-T2_LUND_LOCALGROUPDISK to Aurora]
Line 3: Line 3:
 The recommended mode to move data is using the special ''​fs2-hep''​ storage node. The recommended mode to move data is using the special ''​fs2-hep''​ storage node.
 The SLURM method described below is deprecated, to be used only if the other method doesn'​t work. The SLURM method described below is deprecated, to be used only if the other method doesn'​t work.
 +
 +**SSH** to the //fs2-hep// server from Aurora frontend and start //screen// as described in [[aurora_cluster:​moving_data|Moving data to and from the cluster]] to perform the steps below. //Screen// will ensure that your transfer is not lost if you disconnect.
  
 ===== Using GRID tools on ''​fs2-hep''​ (recommended) ===== ===== Using GRID tools on ''​fs2-hep''​ (recommended) =====
Line 28: Line 30:
 One should interact with the storage using Rucio. One should interact with the storage using Rucio.
  
 +  - Start [[it_tips:​screen]] or ''​byobu''​. The transfer might take long time.
   - Setup the ATLAS environment using ''​cvmfs''​ as explained in [[aurora_cluster:​running_on_aurora#​cvmfs]]   - Setup the ATLAS environment using ''​cvmfs''​ as explained in [[aurora_cluster:​running_on_aurora#​cvmfs]]
   - Run: <code bash>​lsetup rucio</​code>​   - Run: <code bash>​lsetup rucio</​code>​
Line 42: Line 45:
 ==== Downloading data from SE-SNIC-T2_LUND_LOCALGROUPDISK to Aurora ==== ==== Downloading data from SE-SNIC-T2_LUND_LOCALGROUPDISK to Aurora ====
  
-  * Select ​dataset to download: <code bash>​rucio list-datasets-rse SE-SNIC-T2_LUND_LOCALGROUPDISK | less</​code>​+  * Get list of the datasets present in our storage:<code bash>​rucio list-datasets-rse SE-SNIC-T2_LUND_LOCALGROUPDISK | less</​code> ​to identify ​ the **name** of a dataset to download.
   * Select a folder in fs3-hep or fs4-hep as a destination target to download, for example ''/​projects/​hep/​fs4/​scratch/<​yourusername>/​test''​   * Select a folder in fs3-hep or fs4-hep as a destination target to download, for example ''/​projects/​hep/​fs4/​scratch/<​yourusername>/​test''​
  
aurora_cluster/moving_data/grid.1501515865.txt.gz ยท Last modified: 2017/07/31 17:44 by florido