aurora_cluster:moving_data
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
aurora_cluster:moving_data [2017/07/31 18:04] florido |
aurora_cluster:moving_data [2020/08/20 17:13] (current) florido [Using the downloader node] |
||
|---|---|---|---|
| Line 15: | Line 15: | ||
| Small files, whose size are below 10GB, can be moved using aurora's frontend. | Small files, whose size are below 10GB, can be moved using aurora's frontend. | ||
| - | Users of **Mathematical Physics**, as well as any other Lunarc user, can use their favorite tool to download and upload either from your own workstation, the Aurora front-end or the Aurora computing nodes. | + | Users of **Mathematical Physics**, as well as any other Lunarc user, can use their favorite tool to download and upload either from your own workstation or laptop to the Aurora front-end or the Aurora computing nodes. |
| You can read about some of those tools on the [[iridium_cluster:data| Move data to and from the Iridium Cluster]] pages. | You can read about some of those tools on the [[iridium_cluster:data| Move data to and from the Iridium Cluster]] pages. | ||
| Line 33: | Line 33: | ||
| An overview of the upload/download components are shown in the slide below: | An overview of the upload/download components are shown in the slide below: | ||
| - | {{ :aurora_cluster:lunarc_atlas_storage_sketch.png?600 |}} | + | {{ :aurora_cluster:lunarc_atlas_storage_sketch.png?600 | Overview of upload/download on Aurora-HEP}} |
| source: https://docs.google.com/presentation/d/1agBLlMrMe3Pu1RGou5ut5LE0dgzeXFGztKu4Gjn_QBE/edit?usp=sharing | source: https://docs.google.com/presentation/d/1agBLlMrMe3Pu1RGou5ut5LE0dgzeXFGztKu4Gjn_QBE/edit?usp=sharing | ||
| Line 44: | Line 44: | ||
| - The download destination MUST be one of the ''/projects/hep/'' folders or your home folder. All other folders are not writeable by your user. Everything in ''/tmp'' will be deleted regularly. | - The download destination MUST be one of the ''/projects/hep/'' folders or your home folder. All other folders are not writeable by your user. Everything in ''/tmp'' will be deleted regularly. | ||
| - | The picture below shows the various steps. {{ :aurora_cluster:datamovementflowchartbiggfx.png?600 |}} | + | The picture below shows the various steps. {{ :aurora_cluster:datamovementflowchartbiggfx.png?600 | Visual representation of the algorithm in this page}} |
| - | ==== Uploading/Downloading data to/from an external source from Aurora ==== | + | ==== A. Uploading/Downloading data to/from an external source from Aurora ==== |
| - Use your favourite download software. Some suggestions are available at [[iridium_cluster:data|Moving data to and from Iridium]]. If the tool is not available please ask Florido to install it on ''fs2-hep''. | - Use your favourite download software. Some suggestions are available at [[iridium_cluster:data|Moving data to and from Iridium]]. If the tool is not available please ask Florido to install it on ''fs2-hep''. | ||
| - Use your home folder or one of the ''/projects/hep'' folders as a destination folder. Any other path is not writable by your user. The ''/tmp'' folder will be deleted regularly so you should not use that. ''/projects/hep/fs2'' is accessible by everyone, while ''/projects/hep/fs3'' and ''/projects/hep/fs4'' is dedicated storage for the ATLAS project. | - Use your home folder or one of the ''/projects/hep'' folders as a destination folder. Any other path is not writable by your user. The ''/tmp'' folder will be deleted regularly so you should not use that. ''/projects/hep/fs2'' is accessible by everyone, while ''/projects/hep/fs3'' and ''/projects/hep/fs4'' is dedicated storage for the ATLAS project. | ||
| - | ==== Downloading/Uploading data to/from the GRID from Aurora ==== | + | ==== B. Downloading/Uploading data to/from the GRID from Aurora ==== |
| For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described at the link below, so that you can access them from all over the world in the fastest way possible. | For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described at the link below, so that you can access them from all over the world in the fastest way possible. | ||
| Please read the dedicated page [[aurora_cluster:moving_data:grid|Moving data between GRID and Aurora]] | Please read the dedicated page [[aurora_cluster:moving_data:grid|Moving data between GRID and Aurora]] | ||
aurora_cluster/moving_data.1501524266.txt.gz ยท Last modified: 2017/07/31 18:04 by florido
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
