I installed my SSH private key in ~/.ssh/id_rsa and set its permissions to 0600. When I connect to an SSH server which uses my private key in Terminal.app via ssh, a dialog pops up and asks me to enter my password to access the id_rsa file:

I see the same dialog when I connect to an FTP server with the Interarchy GUI client.
Update: I see this dialog every time I connect regardless of whether I check "Remember password in my keychain". It appears two more times if the OK button is clicked regardless of what is entered in the password field.
When I relax these permissions to, say, 0640, I no longer see a dialog asking me for my password but ssh aborts with the following error:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0640 for '/Users/myusername/.ssh/id_rsa' are too open. It is recommended that your private key files are NOT accessible by others. This private key will be ignored. bad permissions: ignore key: /Users/myusername/.ssh/id_rsa
I find the password dialog extremely annoying and I am sure there must be some way to avoid having to dismiss this dialog SSH needs to access the id_rsa file.
Note: I am running Mac OS X 10.6.8.
chmod 600(instead of 644) for it to work – kangax Jul 13 '15 at 17:00ssh-add -Kresolved my issue – Spechal Sep 28 '16 at 15:43ssh-add -Kfixed my issue. Thanks! Not sure what stopped it working in the first place. – Space Dec 21 '16 at 11:00/usr/bin/ssh-add -K ~/.ssh/mykey– n1000 Feb 13 '17 at 20:52