User Tools

Site Tools


aurora_cluster:storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
aurora_cluster:storage [2017/04/12 10:39]
florido [Dedicated storage for Particle Physics]
aurora_cluster:storage [2018/09/11 12:45] (current)
florido [Dedicated storage for Particle Physics]
Line 10: Line 10:
 Please follow these rules to keep the storage tidy. Please follow these rules to keep the storage tidy.
  
-  - Whenever you need to move data to one of the shared folders below,\\ **create a folder with your username** and put the data in there.\\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it. +  - Whenever you need to move data to one of the shared folders below,\\ **create a folder with a name than can be understood by everyone** and put the data in there
-    - Example: <​code:​bash>#​ create folder +  - If the data has been generated by you or if it is for your exclusive use, **create a folder with your username** to accomodate the data. \\ In this way you keep your data away from unwanted people and you can set all the permissions you want on it.
-[pflorido@aurora1 hep]$ mkdir /​projects/​hep/​nobackup/​shared/​pp/​pflorido +
-# copy data +
-[pflorido@aurora1 hep]$ cp mydata ​ /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# list file permissions +
-[pflorido@aurora1 ~]$ ls -ltrah /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
--rw-r--r--. 1 pflorido hep 0 12 apr 12.32 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# change to read/write only user and group (no rights to everyone) +
-[pflorido@aurora1 ~]$ chmod 660 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
-# check applied permissions +
-[pflorido@aurora1 ~]$ ls -ltrah /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata +
--rw-rw----. 1 pflorido hep 0 12 apr 12.32 /​projects/​hep/​nobackup/​shared/​pp/​pflorido/​mydata</​code> ​+
     * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed.     * Data found outside a folder named after a user it is a good candidate to be deleted. Of course there can be exceptions when needed.
   - Remove data that you don't use, for example from the ''​scratch''​ folders. Be kind to your colleagues! storage has a cost.   - Remove data that you don't use, for example from the ''​scratch''​ folders. Be kind to your colleagues! storage has a cost.
-===== Nuclear, Particle and Theorerical ​Physics =====+  - Follow the instructions at [[aurora_cluster:​moving_data|Moving Data to and from the Aurora cluster]] when you start moving data around. 
 +===== Nuclear, Particle and Theoretical ​Physics =====
  
 ==== Dedicated Shared storage server on Aurora ==== ==== Dedicated Shared storage server on Aurora ====
Line 47: Line 37:
 ^ ::: | ::: | ::: | ''/​pp''​ for Particle Physics users | ^ ::: | ::: | ::: | ''/​pp''​ for Particle Physics users |
 ^ ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users | ^ ::: | ::: | ::: | ''/​tp''​ for Theoretical Physics users |
-^ users | Copy of Iridium User homes | This folder contains a copy of each user's private home folder **on Iridium**. **DO NOT use this on Aurora**. This folder will be used as a backup until we plan some other Iridium backup. | /<​username>​ each user her own folder| 
-^ ::: | ::: | ::: | ''/​npguests/<​username>''​ for Nuclear Physics guests | 
-^ ::: | ::: | ::: | ''/​ppguests/<​username>''​ for Particle Physics guests | 
  
  
Line 59: Line 46:
 ==== Dedicated storage for Particle Physics ==== ==== Dedicated storage for Particle Physics ====
  
-Particle Physics can share data to the Iridium Cluster and has two additional storage servers listed below for their exclusive use.+Particle Physics can share data to the [[:​iridium_cluster|Iridium Cluster]] and has two additional storage servers listed below for their exclusive use.
  
 Users not belonging to the division of Particle Physics might be warned to remove their data from those storages, unless otherwise agreed with the Particle Physics division. Users not belonging to the division of Particle Physics might be warned to remove their data from those storages, unless otherwise agreed with the Particle Physics division.
  
-==== Storage on Iridium ​====+=== Storage on Iridium ===
  
 The Iridium storage is still accessible at The Iridium storage is still accessible at
Line 71: Line 58:
 Please refer to the documentation at [[iridium_cluster:​basic_information]] for it. Please refer to the documentation at [[iridium_cluster:​basic_information]] for it.
  
-==== Additional dedicated storage ​====+=== Additional dedicated storage ===
  
-There are two storage servers installed for the exclusive use of particle physics.+There are two storage servers installed for the **exclusive use of Particle Physics**.
 The recommendation is to use these instead of ''/​project/​hep/​nobackup''​ so to give space to the other users of it. The recommendation is to use these instead of ''/​project/​hep/​nobackup''​ so to give space to the other users of it.
  
 :!: ''/​project/​hep/​fs2''​ is a symbolic link to ''/​project/​hep/​nobackup''​ :!: ''/​project/​hep/​fs2''​ is a symbolic link to ''/​project/​hep/​nobackup''​
  
-^ Storage ​path ^ Intended usage ^ Size +^ Server ^ Size (Max) ^ Storage ​paths ^ Intended usage ^ Notes ^ Project ^ Remarks ​
-| ''/​project/​hep/​fs3/​shared''​ | data that is meant to stay for long time | 35TB +| fs3 | 35TB  ​| ''/​projects/​hep/​fs3/​shared''​ | data that is meant to stay for long time | Very fast disk suitable for calculations | ATLAS | Ask Caterina and Will **before** using this space. ​
-| ''/​project/​hep/​fs4/​scratch''​ | temporary data | 35TB |+| fs4 | 35TB  ​| ''/​projects/​hep/​fs4/​scratch''​ | temporary data | Very fast disk suitable for calculations ​ATLAS | Ask Caterina and Will **before** using this space. | 
 +| fs5 | 125TB | ''/​projects/​hep/​fs5/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | Ask Caterina and Will **before** using this space. | 
 +| ::: | :::   | ''/​projects/​hep/​fs5/​scratch''​ | temporary data | ::: | ::: | ::: | 
 +| fs6 | 125TB | ''/​projects/​hep/​fs6/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | ATLAS | Ask Caterina and Will **before** using this space. | 
 +| ::: | :::   | ''/​projects/​hep/​fs6/​scratch''​ | temporary data | ::: | ::: | ::: | 
 +| fs7 | 107TB | ''/​projects/​hep/​fs7/​shared''​ | data that is meant to stay for long time | Slow disk suitable for low read/write frequency | LDMX | | 
 +| ::: | :::   | ''/​projects/​hep/​fs7/​scratch''​ | temporary data | ::: | ::: | ::: | 
 + 
 +=== GRID storage === 
 + 
 +Lunarc hosts a GRID storage for ATLAS. Currently we are migrating from a 20TB storage to a 100TB. 
 + 
 +The local storage name is 
 +   
 +   
 +One can interact through it using the Rucio client.  
 + 
 +Details on how to access this storage are at this page: http://​www.hep.lu.se/​grid/​localgroupdisk.html 
 + 
 +More on how to do transfers at [[aurora_cluster:​moving_data | Moving data to and from the cluster]] 
  
 ===== Mathematical Physics ===== ===== Mathematical Physics =====
  
 At the moment, only the user folders are available. We are negotiating a 10TB common storage with Lunarc. More information is soon to come. At the moment, only the user folders are available. We are negotiating a 10TB common storage with Lunarc. More information is soon to come.
aurora_cluster/storage.1491993555.txt.gz · Last modified: 2017/04/12 10:39 by florido