That's not what I meant. Passphrase on key is fine, Site Manager can be used.
But there are servers out there requiring public key authentication and additionally the traditional username/password auth. Such combination would be impossible in the Site Manager as you can select only one auth type for the server.