how to prevent ZFS from eating too much memory

See this guide:

http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache

 

This syntax is provided starting in the Solaris 10 8/07 release and Nevada (build 51) release.

For example, if an application needs 5 GBytes of memory on a system with 36-GBytes of memory, you could set the arc maximum to 30 GBytes, (0x780000000 or 32212254720 bytes). Set the zfs:zfs_arc_max parameter in the /etc/system file:

set zfs:zfs_arc_max = 0x780000000

or

set zfs:zfs_arc_max = 32212254720