How do I set file.encoding for a junit test in ant?

JUnit supports a child element <jvmarg ...> which should do what you want.

<junit fork="yes"> <jvmarg value="-Dfile.encoding=UTF-8"/> ... </junit>

I assume you were using the fork=yes attribute since this starts a new JVM for the test run, thus the parameters you send into ant at the command line ant -Dfoo=bar do not necessarily propagate to the JVM running the tests.