This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
iridium_cluster:howtos_users [2013/08/27 18:29] florido [Solution 1: scp,sftp,lsftp] |
iridium_cluster:howtos_users [2013/08/27 19:27] florido [Speedup login by using ssh keys] |
||
---|---|---|---|
Line 137: | Line 137: | ||
^ Environment | Script to run | Description | | ^ Environment | Script to run | Description | | ||
^ ATLAS Experiment environment | ''setupATLAS'' | Will setup all the neeeded environment variables for ATLAS experiment, and present a selection of other environments that the user can setup. | | ^ ATLAS Experiment environment | ''setupATLAS'' | Will setup all the neeeded environment variables for ATLAS experiment, and present a selection of other environments that the user can setup. | | ||
+ | |||
+ | ===== Tips'n'Tricks ===== | ||
+ | |||
+ | Suggestions on how to make your life easier when using the cluster. | ||
==== Tips to speedup logging in ==== | ==== Tips to speedup logging in ==== | ||
Line 193: | Line 197: | ||
== References: == | == References: == | ||
* http://sshmenu.sourceforge.net/articles/transparent-mulithop.html | * http://sshmenu.sourceforge.net/articles/transparent-mulithop.html | ||
+ | |||
+ | ==== Speedup login by using ssh keys ==== | ||
+ | |||
+ | An alternative method of authenticating via ssh is by using ssh keys. | ||
+ | |||
+ | :!: ** PLEASE DO NOT USE PASSWORDLESS KEYS. IT IS A GREAT SECURITY RISK. ** :!: | ||
+ | |||
+ | Read about them here: | ||
+ | |||
+ | https://wiki.archlinux.org/index.php/SSH_Keys | ||
+ | |||
+ | |||
+ | ==== How not to loose all your job because you closed a ssh terminal ==== | ||
+ | |||
+ | Use **screen**. //GNU screen// is an amazing tool that opens a remote terminal that is independent on your ssh connection. If the connection drops or you accidentally close the ssh window, it will still run your jobs on the cluster. | ||
+ | |||
+ | A quick and dirty tutorial can be read [[:it_tips#screen|here]], but there's plenty more on the internet. | ||
---- | ---- | ||
Line 199: | Line 220: | ||
Please read the section [[#Common files organization]] before going through this section. | Please read the section [[#Common files organization]] before going through this section. | ||
+ | |||
+ | ==== Rules of thumb ==== | ||
+ | |||
+ | :!: **Please read this carefully.** :!: | ||
+ | |||
+ | When moving data to the shared folders, please follow these common sense rules: | ||
+ | * Create folders for everything you want to share. | ||
+ | * If the data has been produced by you, is nice to create a folder with your name and place everything in it. | ||
+ | * If the data belongs to some specific experiment, dataset or the like, create a folder name that is consistent with that and that is easy for everybody to understand what that is about. | ||
+ | * Don't overdo. Only copy data you/your colleagues need. This is a shared facility. | ||
+ | * Don't remove other user's files unless you advice them and they're ok with it. This is a shared facility. | ||
+ | * Don't expect contents of the ''scratch'' folder to be always there. We still have no policy for that but we will have meetings in which we decide about it. | ||
+ | |||
+ | ==== Data transfer solutions ==== | ||
Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists) | Here's some solutions to move data to the cluster. 1-3 are generic data transfer tools. 4-5 are GRID oriented data transfer tools (mostly for Particle Physicists) | ||
Line 204: | Line 239: | ||
These marked with 8-) are my favourite --- //[[:Florido Paganelli]] 2013/08/27 20:20// | These marked with 8-) are my favourite --- //[[:Florido Paganelli]] 2013/08/27 20:20// | ||
- | ==== Solution 1: scp,sftp,lsftp ==== | + | === Solution 1: scp,sftp,lsftp === |
* **Pros:** | * **Pros:** | ||
Line 210: | Line 245: | ||
* only needs terminal | * only needs terminal | ||
* available almost everywhere | * available almost everywhere | ||
- | * progress bar | + | * progress indicator |
* **Cons:** | * **Cons:** | ||
* not reliable. If connection goes down one must restart the entire transfer. | * not reliable. If connection goes down one must restart the entire transfer. | ||
Line 217: | Line 252: | ||
//Example:// | //Example:// | ||
+ | |||
+ | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' | ||
+ | |||
<code> | <code> | ||
scp ubuntu-12.04.2-desktop-i386.iso n12.iridium:/nfs/shared/pp/ | scp ubuntu-12.04.2-desktop-i386.iso n12.iridium:/nfs/shared/pp/ | ||
</code> | </code> | ||
- | ==== Solution 2: rsync 8-) ==== | + | === Solution 2: rsync === |
+ | |||
+ | 8-) | ||
* **Pros:** | * **Pros:** | ||
Line 228: | Line 268: | ||
* only needs terminal | * only needs terminal | ||
* available on most GNU/Linux platforms | * available on most GNU/Linux platforms | ||
+ | * a bit faster | ||
* **Cons:** | * **Cons:** | ||
* Awkward command line | * Awkward command line | ||
* bad logs | * bad logs | ||
- | * no progress bar | + | * poor progress indicator on many files |
* available on windows but needs special installation | * available on windows but needs special installation | ||
* does **not work** with GRID storage | * does **not work** with GRID storage | ||
- | ==== Solution 3: FileZilla ==== | + | //Example:// |
+ | |||
+ | Moving ''ubuntu-12.04.2-desktop-amd64.iso'' from my local machine to ''n12.iridium'' | ||
+ | |||
+ | <code> | ||
+ | rsync -avz --progress ubuntu-12.04.2-desktop-amd64.iso n12.iridium:/nfs/software/pp/ | ||
+ | </code> | ||
+ | |||
+ | |||
+ | === Solution 3: FileZilla === | ||
* **Pros:** | * **Pros:** | ||
Line 247: | Line 297: | ||
* does **not work** with GRID storage | * does **not work** with GRID storage | ||
- | ==== Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) ==== | + | More about it: https://filezilla-project.org/download.php?type=client |
+ | |||
+ | === Solution 4: NorduGrid ARC tools (arccp, arcls, arcrm) === | ||
* **Pros:** | * **Pros:** | ||
* works with GRID storage | * works with GRID storage | ||
* **Cons:** | * **Cons:** | ||
- | * doesn't work with ATLAS datasets (yet ;) ) | + | * doesn't work with ATLAS datasets (yet ;-) ) |
* uncommon command line interface | * uncommon command line interface | ||
- | ==== Solution 5: dq2 tools ==== | + | //Example:// |
+ | <code> | ||
+ | |||
+ | </code> | ||
+ | |||
+ | |||
+ | === Solution 5: dq2 tools === | ||
* **Pros:** | * **Pros:** | ||
Line 262: | Line 320: | ||
* works with ATLAS datasets | * works with ATLAS datasets | ||
* uncommon command line interface (but some are used to it) | * uncommon command line interface (but some are used to it) | ||
+ | |||
+ | //Example:// | ||
+ | <code> | ||
+ | |||
+ | </code> | ||