aurora_cluster:storage
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
aurora_cluster:storage [2016/11/25 15:23] florido |
aurora_cluster:storage [2020/11/19 09:52] florido [Dedicated division storage] |
||
|---|---|---|---|
| Line 4: | Line 4: | ||
| http://lunarc-documentation.readthedocs.io/en/latest/quick_reference/#file-systems | http://lunarc-documentation.readthedocs.io/en/latest/quick_reference/#file-systems | ||
| - | In particular, the HEP nodes dedicated storage is accessible on hep nodes ''au[193-216]'' at this path: | + | The home folders are maintained and backed up by Lunarc, and they are in a special storage machine dedicated to users. |
| + | |||
| + | 2020-06: I am currently working on some kind of system to keep track of usage and do some minimalistic data management. More information will come. | ||
| + | ===== Basic rules ===== | ||
| + | |||
| + | Please follow these rules to keep the storage tidy. | ||
| + | |||
| + | - Whenever you need to move data to one of the shared folders below,\\ **create a folder with a name than can be understood by everyone** and put the data in there. | ||
| + | - If the data has been generated by you or if it is for your exclusive use, **create a folder with your username** to accomodate the data. \\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it. | ||
| + | * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed. | ||
| + | - Remove data that you don't use, for example from the ''scratch'' folders. Be kind to your colleagues! storage has a cost. | ||
| + | - Follow the instructions at [[aurora_cluster:moving_data|Moving Data to and from the Aurora cluster]] when you start moving data around. | ||
| + | ===== Nuclear, Particle and Theoretical Physics ===== | ||
| + | |||
| + | ==== Dedicated Shared storage server on Aurora ==== | ||
| + | |||
| + | In particular, the HEP nodes dedicated storage usable by members of all divisions is accessible on hep nodes ''au[193-216]'' at this path: | ||
| /projects/hep/nobackup/ | /projects/hep/nobackup/ | ||
| + | | ||
| + | :!: ''/project/hep/fs2'' is a symbolic link to ''/project/hep/nobackup'' | ||
| | | ||
| The organization of data in the above folder is as follows, and users are encouraged to enforce it: | The organization of data in the above folder is as follows, and users are encouraged to enforce it: | ||
| - | ^ Folder name ^ Folder purpose ^ expected filesize ^ Description ^ Subfolders ^ | + | ^ Folder name ^ Folder purpose ^ Description ^ Subfolders ^ |
| - | ^ **software** | Application software | files smaller than 100MB each | This folder hosts software that is not accessible via cvmfs (see later). This usually includes user/project specific libraries and frameworks. | ''/np'' for Nuclear Physics users | | + | ^ **software** | Application software | This folder hosts software that is not accessible via cvmfs (see later). This usually includes user/project specific libraries and frameworks. | ''/np'' for Nuclear Physics users | |
| - | ^ ::: | ::: | ::: | ::: | ''/pp'' for Particle Physics users | | + | ^ ::: | ::: | ::: |''/pp'' for Particle Physics users | |
| - | ^ ::: | ::: | ::: | ::: | ''/tp'' for Theoretical Physics users | | + | ^ ::: | ::: | ::: |''/tp'' for Theoretical Physics users | |
| - | ^ **shared** | Data that will stay for long term | Any file, especially big ones | This folder should be used for long-term stored data. For example, data needed for the whole duration of a phd project or shared among people belonging to the same research group. | ''/np'' for Nuclear Physics users | | + | ^ **shared** | Data that will stay for long term | This folder should be used for long-term stored data. For example, long-term data sets or data needed for the whole duration of a phd project or shared among people belonging to the same research group. | ''/np'' for Nuclear Physics users | |
| - | ^ ::: | ::: | ::: | ::: | ''/pp'' for Particle Physics users | | + | ^ ::: | ::: | ::: |''/pp'' for Particle Physics users | |
| - | ^ ::: | ::: | ::: | ::: | ''/tp'' for Theoretical Physics users | | + | ^ ::: | ::: | ::: | ''/tp'' for Theoretical Physics users | |
| - | ^ **scratch** | Data that will stay for short term | Any file, especially big ones | This folder should be used for short-term stored data. For example, data needed for a week long calculation or temporary calculation. This folder should be considered unreliable as its contents will be purged from time to time. The cleanup interval is yet to be decided | ''/np'' for Nuclear Physics users | | + | ^ **scratch** | Data that will stay for short term | This folder should be used for short-term stored data. For example, data needed for a week long calculation or temporary calculation. This folder should be considered unreliable as its contents will be purged from time to time. The cleanup interval is yet to be decided | ''/np'' for Nuclear Physics users | |
| - | ^ ::: | ::: | ::: | ::: | ''/pp'' for Particle Physics users | | + | ^ ::: | ::: | ::: | ''/pp'' for Particle Physics users | |
| - | ^ ::: | ::: | ::: | ::: | ''/tp'' for Theoretical Physics users | | + | ^ ::: | ::: | ::: | ''/tp'' for Theoretical Physics users | |
| - | ^ users | Copy of Iridium User homes | files smaller than 100MB each | This folder contains a copy of each user's private home folder **on Iridium**. **DO NOT use this on Aurora**. This folder will be used as a backup until we plan some other Iridium backup. | /<username> each user her own folder| | + | |
| - | ^ ::: | ::: | ::: | ::: | ''/npguests/<username>'' for Nuclear Physics guests | | + | |
| - | ^ ::: | ::: | ::: | ::: | ''/ppguests/<username>'' for Particle Physics guests | | + | |
| Line 29: | Line 44: | ||
| ^ ''/cvmfs'' | Special folder containing CERN maintained software | user cannot write | This special folder is dedicated to software provided by CERN. This folder is read-only. Usually the content of this folder are managed via specific scripts that a user can run. If you need to add some software that you cannot find, contact the administrators. | ''/geant4.cern.ch'' for Nuclear Physics users | | ^ ''/cvmfs'' | Special folder containing CERN maintained software | user cannot write | This special folder is dedicated to software provided by CERN. This folder is read-only. Usually the content of this folder are managed via specific scripts that a user can run. If you need to add some software that you cannot find, contact the administrators. | ''/geant4.cern.ch'' for Nuclear Physics users | | ||
| ^ ::: | ::: | ::: | ::: | ''/atlas.cern.ch'' for Particle Physics users | | ^ ::: | ::: | ::: | ::: | ''/atlas.cern.ch'' for Particle Physics users | | ||
| + | |||
| + | === Storage on Iridium === | ||
| The Iridium storage is still accessible at | The Iridium storage is still accessible at | ||
| Line 36: | Line 53: | ||
| Please refer to the documentation at [[iridium_cluster:basic_information]] for it. | Please refer to the documentation at [[iridium_cluster:basic_information]] for it. | ||
| + | ==== Dedicated division storage ==== | ||
| + | |||
| + | Since 2016 we bought several dedicated storages. The ownership, use and permissions of which is described below. | ||
| + | |||
| + | ^ Particle Physics ^^^^^^^ | ||
| + | ^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^ | ||
| + | | fs3 | 35TB | ''/projects/hep/fs3/shared'' | data that is meant to stay for long time | Very fast disk suitable for calculations | ATLAS | Ask Caterina and Will **before** using this space. | | ||
| + | | fs4 | 35TB | ''/projects/hep/fs4/scratch'' | temporary data | Very fast disk suitable for calculations | ATLAS | Ask Caterina and Will **before** using this space. | | ||
| + | | fs5 | 125TB | ''/projects/hep/fs5/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | Ask Caterina and Will **before** using this space. | | ||
| + | | ::: | ::: | ''/projects/hep/fs5/scratch'' | temporary data | ::: | ::: | ::: | | ||
| + | | fs6 | 125TB | ''/projects/hep/fs6/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | Ask Caterina and Will **before** using this space. | | ||
| + | | ::: | ::: | ''/projects/hep/fs6/scratch'' | temporary data | ::: | ::: | ::: | | ||
| + | | fs7 | 107TB | ''/projects/hep/fs7/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | LDMX | Ask Ruth **before** using this space. | | ||
| + | | ::: | ::: | ''/projects/hep/fs7/scratch'' | temporary data | ::: | ::: | ::: | | ||
| + | | fs9 | 350TB | ''/projects/hep/fs9/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | | whole division | | ||
| + | | ::: | ::: | ''/projects/hep/fs9/scratch'' | temporary data | ::: | ::: | ::: | | ||
| + | | ::: | ::: | ''/projects/hep/fs9/users'' | data moved from ''/lunarc/nobackup/users'' | ::: | ::: | ::: | | ||
| + | | SNIC 2020/6-83 | 40TB | ''/lunarc/nobackup/projects/ldmx_simulations'' | Buffer for LDMX data | Slow disk suitable for low read/write frequency | LDMX | Time-limited Storage allocation on Lunarc CentreStorage. The data will be deleted at the end of the project or if the project is not renewed. | | ||
| + | |||
| + | ^ Nuclear Physics ^^^^^^^ | ||
| + | ^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^ | ||
| + | | fs8 | 350TB | ''/projects/hep/fs8/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | | whole division | | ||
| + | | ::: | ::: | ''/projects/hep/fs8/scratch'' | temporary data | ::: | ::: | ::: | | ||
| + | | ::: | ::: | ''/projects/hep/fs8/users'' | data moved from ''/lunarc/nobackup/users'' | ::: | ::: | ::: | | ||
| + | |||
| + | ^ Theoretical Particle Physics ^^^^^^^ | ||
| + | ^ Server ^ Size (Max) ^ Storage paths ^ Intended usage ^ Notes ^ Project ^ Remarks ^ | ||
| + | | fs10 | 350TB | ''/projects/hep/fs10/shared'' | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | | whole division | | ||
| + | | ::: | ::: | ''/projects/hep/fs10/scratch'' | temporary data | ::: | ::: | ::: | | ||
| + | | ::: | ::: | ''/projects/hep/fs10/users'' | data moved from ''/lunarc/nobackup/users'' | ::: | ::: | ::: | | ||
| + | |||
| + | ==== GRID storage ==== | ||
| + | |||
| + | Lunarc hosts a GRID storage for ATLAS. Currently we are migrating from a 20TB storage to a 100TB. | ||
| + | |||
| + | The local storage GRID name is '' SE-SNIC-T2_LUND_LOCALGROUPDISK '' | ||
| + | | ||
| + | One can interact through it using the Rucio client. It cannot be mounted directly on the cluster. Data can only be moved to and fro. Access is restricted to **ATLAS** members. | ||
| + | |||
| + | Details on how to access this storage are at this page: http://www.hep.lu.se/grid/localgroupdisk.html | ||
| + | |||
| + | More on how to do transfers at [[aurora_cluster:moving_data | Moving data to and from the cluster]] | ||
| + | |||
| + | |||
| + | ===== Mathematical Physics ===== | ||
| + | |||
| + | At the moment, only standard LUNARC user folders are available, to my knowledge. Please contact me if you want this information to be updated. | ||
aurora_cluster/storage.txt · Last modified: 2020/11/19 09:52 by florido
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
