How do I set file.encoding for a junit test in ant?

JUnit supports a child element <jvmarg ...> which should do what you want.

<junit 
fork
=
"yes"
>


    <jvmarg
 value
=
"-Dfile.encoding=UTF-8"
/>




  ...



</junit>

I assume you were using the fork=yes attribute since this starts a new JVM for the test run, thus the parameters you send into ant at the command line ant -Dfoo=bar do not necessarily propagate to the JVM running the tests.