Table of Contents

FIXME this page requires a lot of updates.

Moving data between the GRID and Aurora

The recommended mode to move data is using the special fs2-hep storage node. The SLURM method described below is deprecated, to be used only if the other method doesn't work.

SSH to the fs2-hep server from Aurora frontend and start screen as described in Moving data to and from the cluster to perform the steps below. Screen will ensure that your transfer is not lost if you disconnect.

From the GRID point of view, our local GRID storage is a storage element and from ATLAS information system it has a special name, SE-SNIC-T2_LUND_LOCALGROUPDISK . I will use this name in the documentation below. Such storage can be accessed in many ways and protocols, or endpoints, depending on what task you need to carry on. More details about this below. If you already know how datatransfer works in ATLAS, just follow this quick reference card by Oxana:

Otherwise read below.

Prerequisites:

The following has to be done in the order it is presented.

  1. Get a CERN account. Ask your senior team members.
  2. Get a GRID Certificate. Three options (one of the three is enough)
  3. Ask for membership to the ATLAS VOMS server: https://lcg-voms2.cern.ch
    1. :!: IMPORTANT :!: Once your certificate is accepted, check the nickname on your personal voms page.
      This nickname will be used to access Rucio. It is usually the same username as your CERN account login.
  4. Identify the names of the datasets you want to handle (usually the senior person in your research group knows)

Setting up the GRID tools

One should interact with the storage using Rucio.

  1. Start screen or byobu. The transfer might take long time.
  2. Setup the ATLAS environment using cvmfs as explained in cvmfs
  3. Run: 
    lsetup rucio
    1. If asked about the RUCIO_ACCOUNT, make sure to change it to the nickname in the VOMS page in the Prerequisites step.
    2. Follow the onscreen instructions generate a valid atlas GRID proxy
  4. Read the Rucio HowTo, it is hosted by CERN here (only accessible using your CERN account): https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/RucioClientsHowTo and play with rucio commands.
  5. General quick documentation on how to access SE-SNIC-T2_LUND_LOCALGROUPDISK and special endpoint names are explained here: http://www.hep.lu.se/grid/localgroupdisk.html

Downloading data from the grid to SE-SNIC-T2_LUND_LOCALGROUPDISK

This task is about moving the data within GRID storage element. To move a dataset to SE-SNIC-T2_LUND_LOCALGROUPDISK one needs to perform a data transfer request :!: WIP FIXME :!:

Downloading data from SE-SNIC-T2_LUND_LOCALGROUPDISK to Aurora

machine folder path purpose
fs3-hep /projects/hep/fs3/shared/ long term storage of datasets
fs4-hep /projects/hep/fs4/scratch/ short term storage of datasets

Downloading data from any ATLAS SE to Aurora

If you omit the –rse parameter in any of the commands above, rucio will automatically search for the best Storage Element. However the speed and duration of such download is not guaranteed. Please ask the collaboration for their policies regarding such transfers.

Uploading data from Aurora to SE-SNIC-T2_LUND_LOCALGROUPDISK

Before uploading files, it is very important to understand the ATLAS data model. Please read https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/RucioClientsHowTo#Creating_data .

It is recommended to use the user scope unless you're uploading data to be shared with the official collaboration. In that case follow the instructions at the link above. 

rucio list-scopes | grep <your_username>
[pflorido@fs2-hep ~]$ rucio list-scopes | grep fpaganel
user.fpaganel

Remember that dataset names uploaded to rucio are universally unique and cannot be changed in the future, so choose with extreme care.

The generic command for creating a dataset is as follows: 

rucio upload --rse SE-SNIC-T2_LUND_LOCALGROUPDISK user.<username>:<dataset_name> [file1 [file2...]] [/directory]

Where file1, file2, directory are files that will be uploaded inside the dataset called '<dataset name>'

Example: 

[pflorido@fs2-hep ~]$ rucio upload --rse SE-SNIC-T2_LUND_LOCALGROUPDISK user.fpaganel:floridotests ruciotests/downloadfromAU209.txt ruciotests/downloadfromfs2-hep.txt 
2017-07-31 17:21:50,841	WARNING	user.fpaganel:floridotests cannot be distinguished from scope:datasetname. Skipping it.
2017-07-31 17:21:51,469	INFO	Dataset successfully created
2017-07-31 17:21:51,632	INFO	Adding replicas in Rucio catalog
2017-07-31 17:21:51,726	INFO	Replicas successfully added
2017-07-31 17:21:52,872	INFO	File user.fpaganel:downloadfromAU209.txt successfully uploaded on the storage
2017-07-31 17:21:53,174	INFO	Adding replicas in Rucio catalog
2017-07-31 17:21:53,830	INFO	Replicas successfully added
2017-07-31 17:21:55,334	INFO	File user.fpaganel:downloadfromfs2-hep.txt successfully uploaded on the storage
2017-07-31 17:21:57,444	INFO	Will update the file replicas states
2017-07-31 17:21:57,531	INFO	File replicas states successfully updated

Your files are now in the rucio catalog: 

[pflorido@fs2-hep ~]$ rucio list-dids user.fpaganel:*
+----------------------------------------------------+--------------+
| SCOPE:NAME                                         | [DID TYPE]   |
|----------------------------------------------------+--------------|
| user.fpaganel:user.fpaganel.test.0004              | CONTAINER    |
| user.fpaganel:floridotests                         | DATASET      |
| user.fpaganel:user.fpaganel.test.0004.140516165935 | DATASET      |
+----------------------------------------------------+--------------+

Uploading data from Aurora to any other GRID Storage Element

Please read carefully https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/RucioClientsHowTo#Creating_data

Using SLURM Scripts (inefficient, deprecated)

This was a workaround when there was no data mover frontend (fs2-hep). The reason why this technique is discontinued is because it uses the 1GB link of the node to perform the download, while fs2-hep has a dedicated 10GB link just to do the transfers.

Particle Physicists were submitting grid downloads using the SLURM batch system with a script created by Will Kalderon. He kindly provided a link to the scripts:

https://gitlab.cern.ch/lund/useful-scripts/tree/master/downloading

It is mainly intended for Particle Physicist and if you need to access the script you should request clearance to Will. If you see a 404 not found error when clicking the link it means you have no clearance.