User Tools

Site Tools


aurora_cluster:storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
aurora_cluster:storage [2017/04/12 11:05]
florido [Dedicated storage for Particle Physics]
aurora_cluster:storage [2018/09/11 12:34]
florido [Dedicated storage for Particle Physics]
Line 10: Line 10:
 Please follow these rules to keep the storage tidy. Please follow these rules to keep the storage tidy.
  
-  - Whenever you need to move data to one of the shared folders below,\\ **create a folder with your username** and put the data in there.\\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it. +  - Whenever you need to move data to one of the shared folders below,\\ **create a folder with a name than can be understood by everyone** and put the data in there
-    - Example: <​code:​bash>#​ create folder +  - If the data has been generated by you or if it is for your exclusive use, **create a folder with your username** to accomodate the data. \\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it.
-[pflorido@aurora1 hep]$ mkdir /​projects/​hep/​nobackup/​shared/​pp/​pflorido +
-# copy data +
-[pflorido@aurora1 hep]$ cp mydata ​ /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# list file permissions +
-[pflorido@aurora1 ~]$ ls -ltrah /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
--rw-r--r--. 1 pflorido hep 0 12 apr 12.32 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# change to read/write only user and group (no rights to everyone) +
-[pflorido@aurora1 ~]$ chmod 660 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# check applied permissions +
-[pflorido@aurora1 ~]$ ls -ltrah /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
--rw-rw----. 1 pflorido hep 0 12 apr 12.32 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata</​code> ​+
     * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed.     * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed.
   - Remove data that you don't use, for example from the ''​scratch''​ folders. Be kind to your colleagues! storage has a cost.   - Remove data that you don't use, for example from the ''​scratch''​ folders. Be kind to your colleagues! storage has a cost.
-===== Nuclear, Particle and Theorerical ​Physics =====+  - Follow the instructions at [[aurora_cluster:​moving_data|Moving Data to and from the Aurora cluster]] when you start moving data around. 
 +===== Nuclear, Particle and Theoretical ​Physics =====
  
 ==== Dedicated Shared storage server on Aurora ==== ==== Dedicated Shared storage server on Aurora ====
Line 47: Line 37:
 ^ ::: | ::: | ::: | ''/​pp''​ for Particle Physics users | ^ ::: | ::: | ::: | ''/​pp''​ for Particle Physics users |
 ^ ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users | ^ ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users |
-^ users | Copy of Iridium User homes | This folder contains a copy of each user's private home folder **on Iridium**. **DO NOT use this on Aurora**. This folder will be used as a backup until we plan some other Iridium backup. | /<​username>​ each user her own folder| 
-^ ::: | ::: | ::: | ''/​npguests/<​username>''​ for Nuclear Physics guests | 
-^ ::: | ::: | ::: | ''/​ppguests/<​username>''​ for Particle Physics guests | 
  
  
Line 73: Line 60:
 === Additional dedicated storage === === Additional dedicated storage ===
  
-There are two storage servers installed for the exclusive use of particle physics.+There are two storage servers installed for the **exclusive use of Particle Physics**.
 The recommendation is to use these instead of ''/​project/​hep/​nobackup''​ so to give space to the other users of it. The recommendation is to use these instead of ''/​project/​hep/​nobackup''​ so to give space to the other users of it.
  
 :!: ''/​project/​hep/​fs2''​ is a symbolic link to ''/​project/​hep/​nobackup''​ :!: ''/​project/​hep/​fs2''​ is a symbolic link to ''/​project/​hep/​nobackup''​
  
-^ Storage ​path ^ Intended usage ^ Size +^ Server ^ Size (Max) ^ Storage ​paths ^ Intended usage ^ Notes ^ Project ​
-| ''/​project/​hep/​fs3/​shared''​ | data that is meant to stay for long time | 35TB +| fs3 | 35TB  ​| ''/​projects/​hep/​fs3/​shared''​ | data that is meant to stay for long time | Very fast disk suitable for calculations | ATLAS 
-| ''/​project/​hep/​fs4/​scratch''​ | temporary data | 35TB |+| fs4 | 35TB  ​| ''/​projects/​hep/​fs4/​scratch''​ | temporary data | Very fast disk suitable for calculations | ATLAS | 
 +| fs5 | 125TB | ''/​projects/​hep/​fs5/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | 
 +| ::: | :::   | ''/​projects/​hep/​fs5/​scratch''​ | temporary data | ::: | ::: | 
 +| fs6 | 125TB | ''/​projects/​hep/​fs6/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | 
 +| ::: | :::   | ''/​projects/​hep/​fs6/​scratch''​ | temporary data | ::: | ::: | 
 +| fs7 | 107TB | ''/​projects/​hep/​fs7/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | LDMX | 
 +| ::: | :::   | ''/​projects/​hep/​fs7/​scratch''​ | temporary data | ::: | ::: |
  
-==== Moving data to and from the cluster ====+=== GRID storage ​===
  
-The fs2 storage is specialIt is able to download directly ​from the internet with a very fast link without passing through Aurora network, thus freeing some resources for computational tasks.+Lunarc hosts a GRID storage for ATLAS. Currently we are migrating from a 20TB storage to a 100TB. 
 + 
 +The local storage ​name is 
 +   
 +   
 +One can interact through it using the Rucio client 
 + 
 +Details on how to access this storage are at this page: http://​www.hep.lu.se/​grid/​localgroupdisk.html 
 + 
 +More on how to do transfers at [[aurora_cluster:​moving_data | Moving data to and from the cluster]]
  
-The procedure to move data is as follows: 
  
-TO BE WRITTEN 
 ===== Mathematical Physics ===== ===== Mathematical Physics =====
  
 At the moment, only the user folders are available. We are negotiating a 10TB common storage with Lunarc. More information is soon to come. At the moment, only the user folders are available. We are negotiating a 10TB common storage with Lunarc. More information is soon to come.
aurora_cluster/storage.txt · Last modified: 2018/09/11 12:45 by florido