Fork and Exec Using on JAVA

Fork and Exec Using on JAVA

Recently I am reading the codes which write by my others. I think it will cause us problem in the future. It is like a small bomb on our server.

The system is using scala codes as follow to execute shell script
          
          Logger.debug(String.format("Executing script: %s %s %s", "/bin/sh", shellScript, date))
          val process = Runtime.getRuntime().exec(Array("/bin/sh", shellScript, date))
          process.waitFor()

          Logger.debug("Script execution finished, process output.")

          val buf = new StringBuffer
          val infos = Source.fromInputStream(process.getInputStream())(Codec.UTF8).getLines
          val errors = Source.fromInputStream(process.getErrorStream())(Codec.UTF8).getLines

          if (!infos.isEmpty) {
            buf.append("Info:<br/>")
            infos.foreach(buf.append("&nbsp;&nbsp;&nbsp;&nbsp;").append(_).append("<br/>"))
          }
          if (!errors.isEmpty) {
            buf.append("Error:<br/>")
            errors.foreach(buf.append("&nbsp;&nbsp;&nbsp;&nbsp;").append(_).append("<br/>"))
          }

          buf.toString()

Then it will call java.lang.Runtime.java, I am reading the codes from oracle jdk1.7 from my understanding.
    public Process exec(String[] cmdarray, String[] envp, File dir)
        throws IOException {
        return new ProcessBuilder(cmdarray)
            .environment(envp)
            .directory(dir)
            .start();
    }

It will call the method java.lang.ProcessBuilder.java
            return ProcessImpl.start(cmdarray,
                                     environment,
                                     dir,
                                     redirects,
                                     redirectErrorStream);

Open the file java.lang.ProcessImpl.java
        return new UNIXProcess
            (toCString(cmdarray[0]),
             argBlock, args.length,
             envBlock, envc[0],
             toCString(dir),
                 std_fds,
             redirectErrorStream);

Then Open the file is java.lang.UNIXProcess.java, it is the native codes.
    private native int forkAndExec(int mode, byte[] helperpath,
                                   byte[] prog,
                                   byte[] argBlock, int argc,
                                   byte[] envBlock, int envc,
                                   byte[] dir,
                                   int[] fds,
                                   boolean redirectErrorStream)
        throws IOException;

Then I check the man book on linux system from here
http://linux.die.net/man/2/fork
From the document, it is saying that 
Under Linux, fork() is implemented using copy-on-write pages, so the only penalty that it incurs is the time and memory required to duplicate the parent's page tables, and to create a unique task structure for the child.

That means if the parent application is running in 4G memory, this sub process will use 4G too.

Actually our situation is worse
-Xmx6144m -Xmn2048m -XX:PermSize=192m -XX:MaxPermSize=192m

How to check the free memory on my machine
http://blog.scoutapp.com/articles/2010/10/06/determining-free-memory-on-linux

free -m             total       used       free     shared    buffers     cached Mem:          8008       7327        680          0        153        690 -/+ buffers/cache:       6483       1524 Swap:            0          0          0

The actually free memory will be Free(680 MB) + Buffers ( 153 MB) + cached (690 MB) = 1523 MB

On that shell script, we are using shell script only doing the curl(actually it is wget), zip (compress file), ftp related operations. So we have 2 options.

1. Do not use Shell Script

Using Open Source FTP Client http://commons.apache.org/proper/commons-net/
I read the source codes, it is using socket to deal with ftp protocol, there is no shell script there.
http://svn.apache.org/repos/asf/commons/proper/net/trunk/src/main/java/org/apache/commons/net/ftp/FTPClient.java

Using Open Source API to compress the file
http://commons.apache.org/proper/commons-compress/

2. Using a small operation provide restAPI to execute shell command
A standalone app with little memory.


References:
http://blog.youkuaiyun.com/vernonzheng/article/details/8644936
http://linux.die.net/man/2/fork

ftp client
http://commons.apache.org/proper/commons-net/

compress
http://commons.apache.org/proper/commons-compress/