This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
aurora_cluster:moving_data [2017/07/31 18:12] florido [Uploading/Downloading data to/from an external source from Aurora] |
aurora_cluster:moving_data [2020/08/20 17:13] (current) florido [Using the downloader node] |
||
---|---|---|---|
Line 33: | Line 33: | ||
An overview of the upload/download components are shown in the slide below: | An overview of the upload/download components are shown in the slide below: | ||
- | {{ :aurora_cluster:lunarc_atlas_storage_sketch.png?600 |}} | + | {{ :aurora_cluster:lunarc_atlas_storage_sketch.png?600 | Overview of upload/download on Aurora-HEP}} |
source: https://docs.google.com/presentation/d/1agBLlMrMe3Pu1RGou5ut5LE0dgzeXFGztKu4Gjn_QBE/edit?usp=sharing | source: https://docs.google.com/presentation/d/1agBLlMrMe3Pu1RGou5ut5LE0dgzeXFGztKu4Gjn_QBE/edit?usp=sharing | ||
Line 44: | Line 44: | ||
- The download destination MUST be one of the ''/projects/hep/'' folders or your home folder. All other folders are not writeable by your user. Everything in ''/tmp'' will be deleted regularly. | - The download destination MUST be one of the ''/projects/hep/'' folders or your home folder. All other folders are not writeable by your user. Everything in ''/tmp'' will be deleted regularly. | ||
- | The picture below shows the various steps. {{ :aurora_cluster:datamovementflowchartbiggfx.png?600 |}} | + | The picture below shows the various steps. {{ :aurora_cluster:datamovementflowchartbiggfx.png?600 | Visual representation of the algorithm in this page}} |
==== A. Uploading/Downloading data to/from an external source from Aurora ==== | ==== A. Uploading/Downloading data to/from an external source from Aurora ==== | ||
Line 50: | Line 50: | ||
- Use your home folder or one of the ''/projects/hep'' folders as a destination folder. Any other path is not writable by your user. The ''/tmp'' folder will be deleted regularly so you should not use that. ''/projects/hep/fs2'' is accessible by everyone, while ''/projects/hep/fs3'' and ''/projects/hep/fs4'' is dedicated storage for the ATLAS project. | - Use your home folder or one of the ''/projects/hep'' folders as a destination folder. Any other path is not writable by your user. The ''/tmp'' folder will be deleted regularly so you should not use that. ''/projects/hep/fs2'' is accessible by everyone, while ''/projects/hep/fs3'' and ''/projects/hep/fs4'' is dedicated storage for the ATLAS project. | ||
- | ==== Downloading/Uploading data to/from the GRID from Aurora ==== | + | ==== B. Downloading/Uploading data to/from the GRID from Aurora ==== |
For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described at the link below, so that you can access them from all over the world in the fastest way possible. | For resources that can be stored on the GRID, you should definitely stage them on the Lund GRID storage instead, a few ways described at the link below, so that you can access them from all over the world in the fastest way possible. | ||
Please read the dedicated page [[aurora_cluster:moving_data:grid|Moving data between GRID and Aurora]] | Please read the dedicated page [[aurora_cluster:moving_data:grid|Moving data between GRID and Aurora]] |