Why do I need a passphrase?

Passwords aren't very secure, you already know this. If you use one that's easy to remember, it's easier to guess or brute-force. If you use one that's random, it's hard to remember and thus you're more inclined to write the password down. Both of these are Very Bad Things™. This is why you're using ssh keys.

But using a key without a passphrase is basically the same as writing down that random password in a file on your computer. Anyone who gains access to your drive has gained access to every system you use that key with. This is also a Very Bad Thing™. The solution is obvious, add a passphrase.