User Tools

Site Tools


aurora_cluster:storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
aurora_cluster:storage [2016/11/25 15:23]
florido
aurora_cluster:storage [2020/11/19 09:52]
florido [Dedicated division storage]
Line 4: Line 4:
 http://​lunarc-documentation.readthedocs.io/​en/​latest/​quick_reference/#​file-systems http://​lunarc-documentation.readthedocs.io/​en/​latest/​quick_reference/#​file-systems
  
-In particular, the HEP nodes dedicated storage is accessible on hep nodes ''​au[193-216]''​ at this path:+The home folders are maintained and backed up by Lunarc, and they are in a special storage machine dedicated to users. 
 + 
 +2020-06: I am currently working on some kind of system to keep track of usage and do some minimalistic data management. More information will come. 
 +===== Basic rules ===== 
 + 
 +Please follow these rules to keep the storage tidy. 
 + 
 +  - Whenever you need to move data to one of the shared folders below,\\ **create a folder with a name than can be understood by everyone** and put the data in there. 
 +  - If the data has been generated by you or if it is for your exclusive use, **create a folder with your username** to accomodate the data. \\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it. 
 +    * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed. 
 +  - Remove data that you don't use, for example from the ''​scratch''​ folders. Be kind to your colleagues! storage has a cost. 
 +  - Follow the instructions at [[aurora_cluster:​moving_data|Moving Data to and from the Aurora cluster]] when you start moving data around. 
 +===== Nuclear, Particle and Theoretical Physics ===== 
 + 
 +==== Dedicated Shared storage server on Aurora ==== 
 + 
 +In particular, the HEP nodes dedicated storage ​usable by members of all divisions ​is accessible on hep nodes ''​au[193-216]''​ at this path:
  
   /​projects/​hep/​nobackup/​   /​projects/​hep/​nobackup/​
 +  ​
 +:!: ''/​project/​hep/​fs2''​ is a symbolic link to ''/​project/​hep/​nobackup''​
   ​   ​
 The organization of data in the above folder is as follows, and users are encouraged to enforce it: The organization of data in the above folder is as follows, and users are encouraged to enforce it:
  
-^ Folder name ^  Folder purpose ​^ expected filesize ​^ Description ^ Subfolders ^ +^ Folder name ^  Folder purpose ^ Description ^ Subfolders ^ 
-^ **software** | Application software  ​| files smaller than 100MB each | This folder hosts software that is not accessible via cvmfs (see later). This usually includes user/​project specific libraries and frameworks. | ''/​np''​ for Nuclear Physics users | +^ **software** | Application software ​ | This folder hosts software that is not accessible via cvmfs (see later). This usually includes user/​project specific libraries and frameworks. | ''/​np''​ for Nuclear Physics users | 
-::: | ::: | ::: | ::: | ''/​pp''​ for Particle Physics users | +^ ::: | ::: | ::: |''/​pp''​ for Particle Physics users | 
-::: | ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users | +^ ::: | ::: | ::: |''/​tp''​ for Theoretical Physics users | 
-^ **shared** | Data that will stay for long term | Any file, especially big ones | This folder should be used for long-term stored data. For example, data needed for the whole duration of a phd project or shared among people belonging to the same research group. | ''/​np''​ for Nuclear Physics users | +^ **shared** | Data that will stay for long term | This folder should be used for long-term stored data. For example, ​long-term data sets or data needed for the whole duration of a phd project or shared among people belonging to the same research group. | ''/​np''​ for Nuclear Physics users | 
-::: | ::: | ::: | ::: | ''/​pp''​ for Particle Physics users | +^ ::: | ::: | ::: |''/​pp''​ for Particle Physics users | 
-::: | ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users | +^ ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users | 
-^ **scratch** | Data that will stay for short term | Any file, especially big ones | This folder should be used for short-term stored data. For example, data needed for a week long calculation or temporary calculation. This folder should be considered unreliable as its contents will be purged from time to time. The cleanup interval is yet to be decided | ''/​np''​ for Nuclear Physics users | +^ **scratch** | Data that will stay for short term | This folder should be used for short-term stored data. For example, data needed for a week long calculation or temporary calculation. This folder should be considered unreliable as its contents will be purged from time to time. The cleanup interval is yet to be decided | ''/​np''​ for Nuclear Physics users | 
-::: | ::: | ::: | ::: | ''/​pp''​ for Particle Physics users | +^ ::: | ::: | ::: | ''/​pp''​ for Particle Physics users | 
-::: | ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users +^ ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users |
-^ users | Copy of Iridium User homes | files smaller than 100MB each | This folder contains a copy of each user's private home folder **on Iridium**. **DO NOT use this on Aurora**. This folder will be used as a backup until we plan some other Iridium backup. | /<​username>​ each user her own folder| +
-^ ::: | ::: | ::: | ::: | ''/​npguests/<​username>''​ for Nuclear Physics guests | +
-^ ::: | ::: | ::: | ::: | ''/​ppguests/<​username>''​ for Particle Physics guests ​|+
  
  
Line 29: Line 44:
 ^ ''/​cvmfs''​ | Special folder containing CERN maintained software | user cannot write | This special folder is dedicated to software provided by CERN. This folder is read-only. Usually the content of this folder are managed via specific scripts that a user can run. If you need to add some software that you cannot find, contact the administrators. | ''/​geant4.cern.ch''​ for Nuclear Physics users | ^ ''/​cvmfs''​ | Special folder containing CERN maintained software | user cannot write | This special folder is dedicated to software provided by CERN. This folder is read-only. Usually the content of this folder are managed via specific scripts that a user can run. If you need to add some software that you cannot find, contact the administrators. | ''/​geant4.cern.ch''​ for Nuclear Physics users |
 ^ ::: | ::: | ::: | ::: | ''/​atlas.cern.ch''​ for Particle Physics users | ^ ::: | ::: | ::: | ::: | ''/​atlas.cern.ch''​ for Particle Physics users |
 +
 +=== Storage on Iridium ===
  
 The Iridium storage is still accessible at The Iridium storage is still accessible at
Line 36: Line 53:
 Please refer to the documentation at [[iridium_cluster:​basic_information]] for it. Please refer to the documentation at [[iridium_cluster:​basic_information]] for it.
  
 +==== Dedicated division storage ====
 +
 +Since 2016 we bought several dedicated storages. The ownership, use and permissions of which is described below.
 +
 +^ Particle Physics ^^^^^^^
 +^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^
 +| fs3 | 35TB  | ''/​projects/​hep/​fs3/​shared''​ | data that is meant to stay for long time | Very fast disk suitable for calculations | ATLAS | Ask Caterina and Will **before** using this space. |
 +| fs4 | 35TB  | ''/​projects/​hep/​fs4/​scratch''​ | temporary data | Very fast disk suitable for calculations | ATLAS | Ask Caterina and Will **before** using this space. |
 +| fs5 | 125TB | ''/​projects/​hep/​fs5/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | Ask Caterina and Will **before** using this space. |
 +| ::: | :::   | ''/​projects/​hep/​fs5/​scratch''​ | temporary data | ::: | ::: | ::: |
 +| fs6 | 125TB | ''/​projects/​hep/​fs6/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | Ask Caterina and Will **before** using this space. |
 +| ::: | :::   | ''/​projects/​hep/​fs6/​scratch''​ | temporary data | ::: | ::: | ::: |
 +| fs7 | 107TB | ''/​projects/​hep/​fs7/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | LDMX | Ask Ruth **before** using this space. |
 +| ::: | :::   | ''/​projects/​hep/​fs7/​scratch''​ | temporary data | ::: | ::: | ::: | 
 +| fs9 | 350TB | ''/​projects/​hep/​fs9/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency |  | whole division |
 +| ::: | :::   | ''/​projects/​hep/​fs9/​scratch''​ | temporary data | ::: | ::: | ::: | 
 +| ::: | :::   | ''/​projects/​hep/​fs9/​users''​ | data moved from ''/​lunarc/​nobackup/​users''​ | ::: | ::: | ::: | 
 +| SNIC 2020/6-83 | 40TB | ''/​lunarc/​nobackup/​projects/​ldmx_simulations''​ | Buffer for LDMX data | Slow disk suitable for low read/write frequency | LDMX | Time-limited Storage allocation on Lunarc CentreStorage. The data will be deleted at the end of the project or if the project is not renewed. |
 +
 +^ Nuclear Physics ^^^^^^^
 +^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^
 +| fs8 | 350TB | ''/​projects/​hep/​fs8/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency |  | whole division |
 +| ::: | :::   | ''/​projects/​hep/​fs8/​scratch''​ | temporary data | ::: | ::: | ::: | 
 +| ::: | :::   | ''/​projects/​hep/​fs8/​users''​ | data moved from ''/​lunarc/​nobackup/​users''​ | ::: | ::: | ::: | 
 +
 +^ Theoretical Particle Physics ^^^^^^^
 +^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^
 +| fs10 | 350TB | ''/​projects/​hep/​fs10/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency |  | whole division |
 +| ::: | :::   | ''/​projects/​hep/​fs10/​scratch''​ | temporary data | ::: | ::: | ::: | 
 +| ::: | :::   | ''/​projects/​hep/​fs10/​users''​ | data moved from ''/​lunarc/​nobackup/​users''​ | ::: | ::: | ::: | 
 +
 +==== GRID storage ====
 +
 +Lunarc hosts a GRID storage for ATLAS. Currently we are migrating from a 20TB storage to a 100TB.
 +
 +The local storage GRID name is  ''​ SE-SNIC-T2_LUND_LOCALGROUPDISK ''​
 +  ​
 +One can interact through it using the Rucio client. It cannot be mounted directly on the cluster. Data can only be moved to and fro. Access is restricted to **ATLAS** members.
 +
 +Details on how to access this storage are at this page: http://​www.hep.lu.se/​grid/​localgroupdisk.html
 +
 +More on how to do transfers at [[aurora_cluster:​moving_data | Moving data to and from the cluster]]
 +
 +
 +===== Mathematical Physics =====
 +
 +At the moment, only standard LUNARC user folders are available, to my knowledge. Please contact me if you want this information to be updated.
aurora_cluster/storage.txt · Last modified: 2020/11/19 09:52 by florido

Accessibility Statement