aurora_cluster:moving_data
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision | Next revision Both sides next revision | ||
|
aurora_cluster:moving_data [2017/07/31 17:45] florido [Uploading/Downloading data to/from Aurora from your laptop or workstation] |
aurora_cluster:moving_data [2017/07/31 17:47] florido [Uploading/Downloading data to/from Aurora from your laptop or workstation] |
||
|---|---|---|---|
| Line 44: | Line 44: | ||
| ==== Uploading/Downloading data to/from Aurora from your laptop or workstation ==== | ==== Uploading/Downloading data to/from Aurora from your laptop or workstation ==== | ||
| - | For small files (order of tens of gigabytes), small data rates (slow transfers) this can be done via Aurora frontend, you don't need to use ''fs2-hep''. For example, from your laptop: <code bash>''sftp myfile aurora.lunarc.lu.se:/projects/hep/fs2/shared/np/myfolder/myfile'' </code> | + | **This can be done only for small files** (order of tens of gigabytes), small data rates (slow transfers). You don't need to use ''fs2-hep'', but you can go through Aurora's frontend. the For example, from your laptop: <code bash>sftp myfile aurora.lunarc.lu.se:/projects/hep/fs2/shared/np/myfolder/myfile</code> |
| - | For big files (Hundreds of gigabytes up) you should use ''fs2-hep''. Aurora is not a storage facility, therefore is not meant to be accessed by external sources to do data movement. If you do this via the Aurora frontend it is extremely slow and will slow down your colleagues work. Also, Aurora frontend managers might interrupt your transfers if they see it is taking too much time. I strongly recommend to follow the instructions at [[#Uploading/Downloading data to/from an external source from Aurora]] above instead, and eventually run an ssh/ftp server on your own laptop or workstation, or ask the sysadmin for a convenient form of online storage. | + | For big files (Hundreds of gigabytes up) you should use ''fs2-hep'' as described below. Aurora is not a storage facility, therefore is not meant to be accessed by external sources to do data movement. If you move big data via the Aurora frontend it is extremely slow and will slow down your colleagues work. Also, Aurora frontend managers might interrupt your transfers if they see it is taking too much time. I strongly recommend to follow the instructions at [[#Uploading/Downloading data to/from an external source from Aurora]] above instead, and eventually run an ssh/ftp server on your own laptop or workstation, or ask the sysadmin for a convenient form of online storage. |
| For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described under [[#Downloading/Uploading data to/from the GRID to Aurora]], so that you can access them from all over the world in the fastest way possible. | For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described under [[#Downloading/Uploading data to/from the GRID to Aurora]], so that you can access them from all over the world in the fastest way possible. | ||
aurora_cluster/moving_data.txt ยท Last modified: 2020/08/20 17:13 by florido
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
