User Tools

Site Tools


aurora_cluster:moving_data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
aurora_cluster:moving_data [2017/07/31 16:30]
florido
aurora_cluster:moving_data [2020/08/20 17:12]
florido [Moving big data for users of Nuclear, Theoretical and Particle Physics]
Line 10: Line 10:
   * Don't remove other user's files unless you advice them and they'​re ok with it. This is a shared facility.   * Don't remove other user's files unless you advice them and they'​re ok with it. This is a shared facility.
   * Don't expect contents of any ''​scratch''​ folder to be always there. At the moment, however, there is no deletion policy for that.   * Don't expect contents of any ''​scratch''​ folder to be always there. At the moment, however, there is no deletion policy for that.
-===== Moving data for users of Mathematical Physics and generic Lunarc users ===== 
  
-Users of **Mathematical Physics**, as well as any other Lunarc user, can use their favorite tool to download and upload either from your own workstationthe Aurora front-end or the Aurora computing nodes. ​+===== Moving small files for users of Mathematical Physics and all Lunarc users ===== 
 + 
 +Small files, whose size are below 10GB, can be moved using aurora'​s frontend. 
 + 
 +Users of **Mathematical Physics**, as well as any other Lunarc user, can use their favorite tool to download and upload either from your own workstation ​or laptop to the Aurora front-end or the Aurora computing nodes. ​
 You can read about some of those tools on the [[iridium_cluster:​data| Move data to and from the Iridium Cluster]] pages. You can read about some of those tools on the [[iridium_cluster:​data| Move data to and from the Iridium Cluster]] pages.
-===== Moving data for users of Nuclear, Theoretical and Particle Physics ===== 
  
-Users of these division can access the special node //fs2-hep// to be used for downloads or uploads.+For example, from your laptop: <code bash>​sftp myfile aurora.lunarc.lu.se:​/projects/hep/fs2/shared/​np/​myfolder/​myfile</​code>​
  
-These users (in particular ​Particle ​and Theorerical ​Physics) might need to download huge amount of data and therefore it was our objective to offload the Lunarc internal network and the usage of computing nodes as mere downloader nodes.+===== Moving big data for users of Nuclear, Theoretical and Particle Physics ​=====
  
-//fs2-hep// has a direct very fast connection to the internet for downloads. ​However, ​incoming connections are rejected, so one should take into account that this node can download and upload but cannot be used as a source ​to retrieve data from OUTSIDE Lunarc. ​More info below.+For big files (hundreds of gigabytes up) you should use ''​fs2-hep''​ as described below. Aurora is not a storage facility, therefore is not meant to be accessed by external sources to do data movement. If you move big data via the Aurora frontend it is extremely slow and will slow down your colleagues work. Also, Aurora frontend managers might interrupt your transfers if they see it is taking too much time.  
 + 
 +If handling big data, I strongly recommend to run an ssh/ftp server on your own laptop or workstation,​ or ask the sysadmin for a convenient form of online storage that can be accessed **from** Aurora as described below. 
 + 
 +//fs2-hep// has a direct very fast 10GB connection to the internet for downloads ​and uploads. 
 + 
 +Users of Particle and Theoretical Physics might need to download huge amount of data and therefore it was our goal to offload the Lunarc internal network and the usage of computing nodes as mere downloader nodes. 
 + 
 +:​!:​**NOTE**:​!:​ **incoming connections ​from the internet to fs2-hep ​are rejected**. This node can download ​FROM and upload ​TO the internet ​but cannot be accessed directly ​as a server ​to retrieve ​or upload ​data from OUTSIDE Lunarc. ​In other words, **it is not possible to directly connect TO** ''​fs2-hep''​ from the internet via ''​sftp''/''​ssh''/''​rsync''​.
  
 An overview of the upload/​download components are shown in the slide below: An overview of the upload/​download components are shown in the slide below:
-{{ :​aurora_cluster:​lunarc_atlas_storage_sketch.png?​600 |}}+{{ :​aurora_cluster:​lunarc_atlas_storage_sketch.png?​600 | Overview of upload/​download on Aurora-HEP}}
 source: https://​docs.google.com/​presentation/​d/​1agBLlMrMe3Pu1RGou5ut5LE0dgzeXFGztKu4Gjn_QBE/​edit?​usp=sharing source: https://​docs.google.com/​presentation/​d/​1agBLlMrMe3Pu1RGou5ut5LE0dgzeXFGztKu4Gjn_QBE/​edit?​usp=sharing
  
Line 29: Line 39:
  
   - Login to //​aurora.lunarc.lu.se// ​   - Login to //​aurora.lunarc.lu.se// ​
-  - Login to ''​fs2-hep'':​ <code bash>ssh fs2-hep+  - Login to ''​fs2-hep'':​ <code bash>ssh fs2-hep</​code>​
   - Start ''​screen'':​ <code bash>​screen</​code>​   - Start ''​screen'':​ <code bash>​screen</​code>​
   - Choose one of the upload/​download methods below.   - Choose one of the upload/​download methods below.
-  - The download destination MUST be one of the ''/​project/​hep/''​ folders or your home folder. All other folders are not writable ​by your user. Everything in ''/​tmp''​ will be deleted regularly. +  - The download destination MUST be one of the ''/​projects/​hep/''​ folders or your home folder. All other folders are not writeable ​by your user. Everything in ''/​tmp''​ will be deleted regularly.
- +
-The picture below shows the various steps. {{ :​aurora_cluster:​datamovementflowchartbiggfx.png?​800 |}} +
-==== Uploading/​Downloading data to/from an external source to Aurora ==== +
-  - Use your favourite download software. Some suggestions are available at [[iridium_cluster:​data|Moving data to and from Iridium]] +
-  - Use your home folder one of the ''/​project/​hep''​ folders as a destination folder. Any other path is not writable by your user. The ''/​tmp''​ folder will be deleted regularly so you should not use that. +
- +
-==== Uploading/​Downloading data to/from Aurora from your laptop or workstation ====+
  
-You should avoid doing thisAurora is not a storage facility, therefore is not meant to be accessed by external sources to do data movement. It is possible to do that through Aurora frontend but this is extremely slow and will slow down your colleagues work. Also, Aurora frontend managers might interrupt your transfers if they see it is taking too much time. I strongly recommend to follow the instructions at [[#​Uploading/​Downloading data to/from an external source to Aurora]] above instead, and eventually run an ssh/ftp server on your own laptop or workstation,​ or ask the sysadmin for a convenient form of online storage.+The picture below shows the various steps{{ :​aurora_cluster:​datamovementflowchartbiggfx.png?600 |}}
  
-For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, ​few ways described under [[#Using GRID tools]], ​so that you can access them from all over the world in the fastest way possible.+==== A. Uploading/​Downloading data to/from an external source from Aurora ==== 
 +  - Use your favourite download software. Some suggestions are available at [[iridium_cluster:​data|Moving data to and from Iridium]]. If the tool is not available please ask Florido to install it on ''​fs2-hep''​. 
 +  - Use your home folder or one of the ''/​projects/​hep''​ folders as destination folder. Any other path is not writable by your user. The ''/​tmp'' ​ folder will be deleted regularly ​so you should not use that. ''/​projects/​hep/​fs2''​ is accessible by everyone, while ''/​projects/​hep/​fs3''​ and ''/​projects/​hep/​fs4''​ is dedicated storage for the ATLAS project.
  
 +==== B. Downloading/​Uploading data to/from the GRID from Aurora ====
  
-==== Downloading/​Uploading data to/​from ​the GRID to Aurora ====+For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described at the link below, so that you can access them from all over the world in the fastest way possible.
  
 Please read the dedicated page [[aurora_cluster:​moving_data:​grid|Moving data between GRID and Aurora]] Please read the dedicated page [[aurora_cluster:​moving_data:​grid|Moving data between GRID and Aurora]]
aurora_cluster/moving_data.txt · Last modified: 2020/08/20 17:13 by florido

Accessibility Statement