User Tools

Site Tools


aurora_cluster:storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
aurora_cluster:storage [2017/04/12 11:05]
florido [Dedicated storage for Particle Physics]
aurora_cluster:storage [2018/01/29 17:54] (current)
florido [Dedicated storage for Particle Physics]
Line 10: Line 10:
 Please follow these rules to keep the storage tidy. Please follow these rules to keep the storage tidy.
  
-  - Whenever you need to move data to one of the shared folders below,\\ **create a folder with your username** and put the data in there.\\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it. +  - Whenever you need to move data to one of the shared folders below,\\ **create a folder with a name than can be understood by everyone** and put the data in there
-    - Example: <​code:​bash>#​ create folder +  - If the data has been generated by you or if it is for your exclusive use, **create a folder with your username** to accomodate the data. \\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it.
-[pflorido@aurora1 hep]$ mkdir /​projects/​hep/​nobackup/​shared/​pp/​pflorido +
-# copy data +
-[pflorido@aurora1 hep]$ cp mydata ​ /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# list file permissions +
-[pflorido@aurora1 ~]$ ls -ltrah /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
--rw-r--r--. 1 pflorido hep 0 12 apr 12.32 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# change to read/write only user and group (no rights to everyone) +
-[pflorido@aurora1 ~]$ chmod 660 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# check applied permissions +
-[pflorido@aurora1 ~]$ ls -ltrah /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
--rw-rw----. 1 pflorido hep 0 12 apr 12.32 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata</​code> ​+
     * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed.     * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed.
   - Remove data that you don't use, for example from the ''​scratch''​ folders. Be kind to your colleagues! storage has a cost.   - Remove data that you don't use, for example from the ''​scratch''​ folders. Be kind to your colleagues! storage has a cost.
-===== Nuclear, Particle and Theorerical ​Physics =====+  - Follow the instructions at [[aurora_cluster:​moving_data|Moving Data to and from the Aurora cluster]] when you start moving data around. 
 +===== Nuclear, Particle and Theoretical ​Physics =====
  
 ==== Dedicated Shared storage server on Aurora ==== ==== Dedicated Shared storage server on Aurora ====
Line 47: Line 37:
 ^ ::: | ::: | ::: | ''/​pp''​ for Particle Physics users | ^ ::: | ::: | ::: | ''/​pp''​ for Particle Physics users |
 ^ ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users | ^ ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users |
-^ users | Copy of Iridium User homes | This folder contains a copy of each user's private home folder **on Iridium**. **DO NOT use this on Aurora**. This folder will be used as a backup until we plan some other Iridium backup. | /<​username>​ each user her own folder| 
-^ ::: | ::: | ::: | ''/​npguests/<​username>''​ for Nuclear Physics guests | 
-^ ::: | ::: | ::: | ''/​ppguests/<​username>''​ for Particle Physics guests | 
  
  
Line 73: Line 60:
 === Additional dedicated storage === === Additional dedicated storage ===
  
-There are two storage servers installed for the exclusive use of particle physics.+There are two storage servers installed for the **exclusive use of Particle Physics**.
 The recommendation is to use these instead of ''/​project/​hep/​nobackup''​ so to give space to the other users of it. The recommendation is to use these instead of ''/​project/​hep/​nobackup''​ so to give space to the other users of it.
  
 :!: ''/​project/​hep/​fs2''​ is a symbolic link to ''/​project/​hep/​nobackup''​ :!: ''/​project/​hep/​fs2''​ is a symbolic link to ''/​project/​hep/​nobackup''​
  
-^ Storage ​path ^ Intended usage ^ Size +^ Server ^ Size (Max) ^ Storage ​paths ^ Intended usage ^ Notes 
-| ''/​project/​hep/​fs3/​shared''​ | data that is meant to stay for long time | 35TB +| fs3 | 35TB  ​| ''/​projects/​hep/​fs3/​shared''​ | data that is meant to stay for long time | Very fast disk suitable for calculations ​
-| ''/​project/​hep/​fs4/​scratch''​ | temporary data | 35TB |+| fs4 | 35TB  ​| ''/​projects/​hep/​fs4/​scratch''​ | temporary data | Very fast disk suitable for calculations | 
 +| fs5 | 125TB | ''/​projects/​hep/​fs5/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | 
 +| ::: | :::   | ''/​projects/​hep/​fs5/​scratch''​ | temporary data | ::: | 
 +| fs6 | 125TB | ''/​projects/​hep/​fs6/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | 
 +| ::: | :::   | ''/​projects/​hep/​fs6/​scratch''​ | temporary data | ::: |
  
-==== Moving data to and from the cluster ====+=== Grid storage ​===
  
-The fs2 storage is specialIt is able to download directly ​from the internet with a very fast link without passing through Aurora network, thus freeing some resources for computational tasks.+Lunarc hosts a grid storage for ATLAS. Currently we are migrating from a 20TB storage to a 100TB. 
 + 
 +The local storage ​name is 
 +   
 +   
 +One can interact through it using the Rucio client 
 + 
 +Details on how to access this storage are at this page: http://​www.hep.lu.se/​grid/​localgroupdisk.html 
 + 
 +More on how to do transfers at [[aurora_cluster:​moving_data | Moving data to and from the cluster]]
  
-The procedure to move data is as follows: 
  
-TO BE WRITTEN 
 ===== Mathematical Physics ===== ===== Mathematical Physics =====
  
 At the moment, only the user folders are available. We are negotiating a 10TB common storage with Lunarc. More information is soon to come. At the moment, only the user folders are available. We are negotiating a 10TB common storage with Lunarc. More information is soon to come.
aurora_cluster/storage.1491995127.txt.gz · Last modified: 2017/04/12 11:05 by florido