This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
aurora_cluster:moving_data [2017/07/31 18:05] florido [Moving small files for users of Mathematical Physics and all Lunarc users] |
aurora_cluster:moving_data [2020/08/20 17:13] (current) florido [Using the downloader node] |
||
---|---|---|---|
Line 33: | Line 33: | ||
An overview of the upload/download components are shown in the slide below: | An overview of the upload/download components are shown in the slide below: | ||
- | {{ :aurora_cluster:lunarc_atlas_storage_sketch.png?600 |}} | + | {{ :aurora_cluster:lunarc_atlas_storage_sketch.png?600 | Overview of upload/download on Aurora-HEP}} |
source: https://docs.google.com/presentation/d/1agBLlMrMe3Pu1RGou5ut5LE0dgzeXFGztKu4Gjn_QBE/edit?usp=sharing | source: https://docs.google.com/presentation/d/1agBLlMrMe3Pu1RGou5ut5LE0dgzeXFGztKu4Gjn_QBE/edit?usp=sharing | ||
Line 44: | Line 44: | ||
- The download destination MUST be one of the ''/projects/hep/'' folders or your home folder. All other folders are not writeable by your user. Everything in ''/tmp'' will be deleted regularly. | - The download destination MUST be one of the ''/projects/hep/'' folders or your home folder. All other folders are not writeable by your user. Everything in ''/tmp'' will be deleted regularly. | ||
- | The picture below shows the various steps. {{ :aurora_cluster:datamovementflowchartbiggfx.png?600 |}} | + | The picture below shows the various steps. {{ :aurora_cluster:datamovementflowchartbiggfx.png?600 | Visual representation of the algorithm in this page}} |
- | ==== Uploading/Downloading data to/from an external source from Aurora ==== | + | ==== A. Uploading/Downloading data to/from an external source from Aurora ==== |
- Use your favourite download software. Some suggestions are available at [[iridium_cluster:data|Moving data to and from Iridium]]. If the tool is not available please ask Florido to install it on ''fs2-hep''. | - Use your favourite download software. Some suggestions are available at [[iridium_cluster:data|Moving data to and from Iridium]]. If the tool is not available please ask Florido to install it on ''fs2-hep''. | ||
- Use your home folder or one of the ''/projects/hep'' folders as a destination folder. Any other path is not writable by your user. The ''/tmp'' folder will be deleted regularly so you should not use that. ''/projects/hep/fs2'' is accessible by everyone, while ''/projects/hep/fs3'' and ''/projects/hep/fs4'' is dedicated storage for the ATLAS project. | - Use your home folder or one of the ''/projects/hep'' folders as a destination folder. Any other path is not writable by your user. The ''/tmp'' folder will be deleted regularly so you should not use that. ''/projects/hep/fs2'' is accessible by everyone, while ''/projects/hep/fs3'' and ''/projects/hep/fs4'' is dedicated storage for the ATLAS project. | ||
- | ==== Downloading/Uploading data to/from the GRID from Aurora ==== | + | ==== B. Downloading/Uploading data to/from the GRID from Aurora ==== |
For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described at the link below, so that you can access them from all over the world in the fastest way possible. | For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described at the link below, so that you can access them from all over the world in the fastest way possible. | ||
Please read the dedicated page [[aurora_cluster:moving_data:grid|Moving data between GRID and Aurora]] | Please read the dedicated page [[aurora_cluster:moving_data:grid|Moving data between GRID and Aurora]] |