Differences

This shows you the differences between two versions of the page.

--- aurora_cluster:moving_data [2017/07/31 15:48]
florido
+++ aurora_cluster:moving_data [2017/07/31 17:47]
florido [Uploading/Downloading data to/from Aurora from your laptop or workstation]
@@ Line 20: / Line 20: @@
 These users (in particular Particle and Theorerical Physics) might need to download huge amount of data and therefore it was our objective to offload the Lunarc internal network and the usage of computing nodes as mere downloader nodes.
-//fs2-hep// has a direct very fast connection to the internet for downloads. However, incoming connections are rejected, so one should take into account that this node can download and upload but cannot be used as a source to retrieve data from OUTSIDE Lunarc. More info below.
+//fs2-hep// has a direct very fast connection to the internet for downloads and uploads.
+:!:**NOTE**:!: **incoming connections from the internet are rejected**. This node can download FROM and upload TO the internet but cannot be accessed directly as a server to retrieve or upload data from OUTSIDE Lunarc. In other words, **it is not possible to directly connect TO** ''fs2-hep'' from the internet via ''sftp''/''ssh''/''rsync''. You can only run those on ''fs2-hep'' itself. Read more about this in [[#Uploading/Downloading data to/from Aurora from your laptop or workstation]].
 An overview of the upload/download components are shown in the slide below:
@@ Line 29: / Line 31: @@
   - Login to //aurora.lunarc.lu.se//
-  - Login to ''fs2-hep'': <code:bash>ssh fs2-hep</code>
+  - Login to ''fs2-hep'': <code bash>ssh fs2-hep</code>
+  - Start ''screen'': <code bash>screen</code>
   - Choose one of the upload/download methods below.
-  - The download destination MUST be one of the ''/project/hep/'' folders or your home folder. All other folders are not writable by your user. Everything in ''/tmp'' will be deleted regularly.
+  - The download destination MUST be one of the ''/projects/hep/'' folders or your home folder. All other folders are not writeable by your user. Everything in ''/tmp'' will be deleted regularly.
-==== Uploading/Downloading data to/from an external source to Aurora ====
+The picture below shows the various steps. {{ :aurora_cluster:datamovementflowchartbiggfx.png?600 |}}
+==== Uploading/Downloading data to/from an external source from Aurora ====
   - Use your favourite download software. Some suggestions are available at [[iridium_cluster:data|Moving data to and from Iridium]]
-  - Use your home folder one of the ''/project/hep'' folders as a destination folder. Any other path is not writable by your user. The ''/tmp'' folder will be deleted regularly so you should not use that.
+  - Use your home folder or one of the ''/projects/hep'' folders as a destination folder. Any other path is not writable by your user. The ''/tmp''  folder will be deleted regularly so you should not use that. ''/projects/hep/fs2'' is accessible by everyone, while ''/projects/hep/fs3'' and ''/projects/hep/fs4'' is dedicated storage for the ATLAS project.
 ==== Uploading/Downloading data to/from Aurora from your laptop or workstation ====
-You should avoid doing this. Aurora is not a storage facility, therefore is not meant to be accessed by external sources to do data movement. It is possible to do that through Aurora frontend but this is extremely slow and will slow down your colleagues work. Also, Aurora frontend managers might interrupt your transfers if they see it is taking too much time. I strongly recommend to follow the instructions at [[#Uploading/Downloading data to/from an external source to Aurora]] above instead, and eventually run an ssh/ftp server on your own laptop or workstation, or ask the sysadmin for a convenient form of online storage.
+**This can be done only for small files** (order of tens of gigabytes), small data rates (slow transfers). You don't need to use ''fs2-hep'', but you can go through Aurora's frontend. the  For example, from your laptop: <code bash>sftp myfile aurora.lunarc.lu.se:/projects/hep/fs2/shared/np/myfolder/myfile</code>
+For big files (Hundreds of gigabytes up) you should use ''fs2-hep'' as described below. Aurora is not a storage facility, therefore is not meant to be accessed by external sources to do data movement. If you move big data via the Aurora frontend it is extremely slow and will slow down your colleagues work. Also, Aurora frontend managers might interrupt your transfers if they see it is taking too much time. I strongly recommend to follow the instructions at [[#Uploading/Downloading data to/from an external source from Aurora]] above instead, and eventually run an ssh/ftp server on your own laptop or workstation, or ask the sysadmin for a convenient form of online storage.
-For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described under [[#Using GRID tools]], so that you can access them from all over the world in the fastest way possible.
+For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described under [[#Downloading/Uploading data to/from the GRID to Aurora]], so that you can access them from all over the world in the fastest way possible.
-==== Downloading/Uploading data to/from the GRID to Aurora ====
+==== Downloading/Uploading data to/from the GRID from Aurora ====
 Please read the dedicated page [[aurora_cluster:moving_data:grid|Moving data between GRID and Aurora]]

pfwiki

User Tools

Site Tools

Differences

Page Tools