This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
aurora_cluster:moving_data:grid [2017/07/31 15:44] florido |
aurora_cluster:moving_data:grid [2017/10/05 08:50] florido [Downloading data from SE-SNIC-T2_LUND_LOCALGROUPDISK to Aurora] |
||
---|---|---|---|
Line 3: | Line 3: | ||
The recommended mode to move data is using the special ''fs2-hep'' storage node. | The recommended mode to move data is using the special ''fs2-hep'' storage node. | ||
The SLURM method described below is deprecated, to be used only if the other method doesn't work. | The SLURM method described below is deprecated, to be used only if the other method doesn't work. | ||
+ | |||
+ | **SSH** to the //fs2-hep// server from Aurora frontend and start //screen// as described in [[aurora_cluster:moving_data|Moving data to and from the cluster]] to perform the steps below. //Screen// will ensure that your transfer is not lost if you disconnect. | ||
===== Using GRID tools on ''fs2-hep'' (recommended) ===== | ===== Using GRID tools on ''fs2-hep'' (recommended) ===== | ||
Line 28: | Line 30: | ||
One should interact with the storage using Rucio. | One should interact with the storage using Rucio. | ||
+ | - Start [[it_tips:screen]] or ''byobu''. The transfer might take long time. | ||
- Setup the ATLAS environment using ''cvmfs'' as explained in [[aurora_cluster:running_on_aurora#cvmfs]] | - Setup the ATLAS environment using ''cvmfs'' as explained in [[aurora_cluster:running_on_aurora#cvmfs]] | ||
- Run: <code bash>lsetup rucio</code> | - Run: <code bash>lsetup rucio</code> | ||
Line 42: | Line 45: | ||
==== Downloading data from SE-SNIC-T2_LUND_LOCALGROUPDISK to Aurora ==== | ==== Downloading data from SE-SNIC-T2_LUND_LOCALGROUPDISK to Aurora ==== | ||
- | * Select a dataset to download: <code bash>rucio list-datasets-rse SE-SNIC-T2_LUND_LOCALGROUPDISK | less</code> | + | * Get a list of the datasets present in our storage:<code bash>rucio list-datasets-rse SE-SNIC-T2_LUND_LOCALGROUPDISK | less</code> to identify the **name** of a dataset to download. |
* Select a folder in fs3-hep or fs4-hep as a destination target to download, for example ''/projects/hep/fs4/scratch/<yourusername>/test'' | * Select a folder in fs3-hep or fs4-hep as a destination target to download, for example ''/projects/hep/fs4/scratch/<yourusername>/test'' | ||