pfwiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Storage on Aurora ======

Storage in the Aurora cluster is organized as detailed by Lunarc. Read the following: 
http://lunarc-documentation.readthedocs.io/en/latest/quick_reference/#file-systems

The home folders are maintained and backed up by Lunarc, and they are in a special storage machine dedicated to users.

2020-06: I am currently working on some kind of system to keep track of usage and do some minimalistic data management. More information will come.
===== Basic rules =====

Please follow these rules to keep the storage tidy.

  - Whenever you need to move data to one of the shared folders below,\\ **create a folder with a name than can be understood by everyone** and put the data in there.
  - If the data has been generated by you or if it is for your exclusive use, **create a folder with your username** to accomodate the data. \\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it.
    * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed.
  - Remove data that you don't use, for example from the ''scratch'' folders. Be kind to your colleagues! storage has a cost.
  - Follow the instructions at [[aurora_cluster:moving_data|Moving Data to and from the Aurora cluster]] when you start moving data around.
===== Nuclear, Particle and Theoretical Physics =====

==== Dedicated Shared storage server on Aurora ====

In particular, the HEP nodes dedicated storage usable by members of all divisions is accessible on hep nodes ''au[193-216]'' at this path:

  /projects/hep/nobackup/
  
:!: ''/project/hep/fs2'' is a symbolic link to ''/project/hep/nobackup''
  
The organization of data in the above folder is as follows, and users are encouraged to enforce it:

^ Folder name ^  Folder purpose ^ Description ^ Subfolders ^
^ **software** | Application software  | This folder hosts software that is not accessible via cvmfs (see later). This usually includes user/project specific libraries and frameworks. | ''/np'' for Nuclear Physics users |
^ ::: | ::: | ::: |''/pp'' for Particle Physics users |
^ ::: | ::: | ::: |''/tp'' for Theoretical Physics users |
^ **shared** | Data that will stay for long term | This folder should be used for long-term stored data. For example, long-term data sets or data needed for the whole duration of a phd project or shared among people belonging to the same research group. | ''/np'' for Nuclear Physics users |
^ ::: | ::: | ::: |''/pp'' for Particle Physics users |
^ ::: | ::: | ::: | ''/tp'' for Theoretical Physics users |
^ **scratch** | Data that will stay for short term | This folder should be used for short-term stored data. For example, data needed for a week long calculation or temporary calculation. This folder should be considered unreliable as its contents will be purged from time to time. The cleanup interval is yet to be decided | ''/np'' for Nuclear Physics users |
^ ::: | ::: | ::: | ''/pp'' for Particle Physics users |
^ ::: | ::: | ::: | ''/tp'' for Theoretical Physics users |


Other special folders:

^ ''/cvmfs'' | Special folder containing CERN maintained software | user cannot write | This special folder is dedicated to software provided by CERN. This folder is read-only. Usually the content of this folder are managed via specific scripts that a user can run. If you need to add some software that you cannot find, contact the administrators. | ''/geant4.cern.ch'' for Nuclear Physics users |
^ ::: | ::: | ::: | ::: | ''/atlas.cern.ch'' for Particle Physics users |

=== Storage on Iridium ===

The Iridium storage is still accessible at

  /projects/hep/iridium/
  
Please refer to the documentation at [[iridium_cluster:basic_information]] for it.

==== Dedicated division storage ====

Since 2016 we bought several dedicated storages. The ownership, use and permissions of which is described below.

^ Particle Physics ^^^^^^^
^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^
| fs3 | 35TB  | ''/projects/hep/fs3/shared'' | data that is meant to stay for long time | Very fast disk suitable for calculations | ATLAS | Ask Caterina and Will **before** using this space. |
| fs4 | 35TB  | ''/projects/hep/fs4/scratch'' | temporary data | Very fast disk suitable for calculations | ATLAS | Ask Caterina and Will **before** using this space. |
| fs5 | 125TB | ''/projects/hep/fs5/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | Ask Caterina and Will **before** using this space. |
| ::: | :::   | ''/projects/hep/fs5/scratch'' | temporary data | ::: | ::: | ::: |
| fs6 | 125TB | ''/projects/hep/fs6/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | Ask Caterina and Will **before** using this space. |
| ::: | :::   | ''/projects/hep/fs6/scratch'' | temporary data | ::: | ::: | ::: |
| fs7 | 107TB | ''/projects/hep/fs7/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | LDMX | Ask Ruth **before** using this space. |
| ::: | :::   | ''/projects/hep/fs7/scratch'' | temporary data | ::: | ::: | ::: | 
| fs9 | 350TB | ''/projects/hep/fs9/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency |  | whole division |
| ::: | :::   | ''/projects/hep/fs9/scratch'' | temporary data | ::: | ::: | ::: | 
| ::: | :::   | ''/projects/hep/fs9/users'' | data moved from ''/lunarc/nobackup/users'' | ::: | ::: | ::: | 
| SNIC 2020/6-83 | 40TB | ''/lunarc/nobackup/projects/ldmx_simulations'' | Buffer for LDMX data | Slow disk suitable for low read/write frequency | LDMX | Time-limited Storage allocation on Lunarc CentreStorage. The data will be deleted at the end of the project or if the project is not renewed. |

^ Nuclear Physics ^^^^^^^
^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^
| fs8 | 350TB | ''/projects/hep/fs8/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency |  | whole division |
| ::: | :::   | ''/projects/hep/fs8/scratch'' | temporary data | ::: | ::: | ::: | 
| ::: | :::   | ''/projects/hep/fs8/users'' | data moved from ''/lunarc/nobackup/users'' | ::: | ::: | ::: | 

^ Theoretical Particle Physics ^^^^^^^
^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^
| fs10 | 350TB | ''/projects/hep/fs10/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency |  | whole division |
| ::: | :::   | ''/projects/hep/fs10/scratch'' | temporary data | ::: | ::: | ::: | 
| ::: | :::   | ''/projects/hep/fs10/users'' | data moved from ''/lunarc/nobackup/users'' | ::: | ::: | ::: | 

==== GRID storage ====

Lunarc hosts a GRID storage for ATLAS. Currently we are migrating from a 20TB storage to a 100TB.

The local storage GRID name is  '' SE-SNIC-T2_LUND_LOCALGROUPDISK ''
  
One can interact through it using the Rucio client. It cannot be mounted directly on the cluster. Data can only be moved to and fro. Access is restricted to **ATLAS** members.

Details on how to access this storage are at this page: http://www.hep.lu.se/grid/localgroupdisk.html

More on how to do transfers at [[aurora_cluster:moving_data | Moving data to and from the cluster]]


===== Mathematical Physics =====

At the moment, only standard LUNARC user folders are available, to my knowledge. Please contact me if you want this information to be updated.
pfwiki

User Tools

Site Tools

Page Tools
