android compile tasks中dex过程源码情景分析-优快云博客

本文详细剖析了Android构建系统中MultiDex机制的工作原理，包括.dex文件的生成过程及secondary dex的创建逻辑。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

0x00 前言

MultiDex中出现的main dex capacity exceeded解决之道中我们知道main dex的class可以由maindexlist.txt指定，Android MultiDex机制杂谈中我们分析了google MultiDex机制中Secondary dex的install过程，那么，我们的app在android gradle build过程中，.dex文件是怎么创建的呢？再者，Secondary dex中的class是按什么顺序分配到不同dex中的呢？

0x01 android build system概述

为了解答上面的两个问题，本文将进一步分析android build system源码。
android build system是google提供的一组用来构建、运行、测试和打包我们app的工具集，包含了aapt、aidl、javac、dex、apkbuilder、Jarsigner、zipalign等工具。在我们构建app时，build进程会去按一定顺序调用上述工具来生成相应文件，而最终的输出将会是一个完整的可安装的.apk文件，构建流程如下：

构建系统先从product flavors, build types和dependencies中合并资源，如果不同目录下有重名资源，将按以下优先级进行覆盖：

dependencies > build types > product flavors > main source directory

aapt编译应用的资源文件（如AndroidManifest.xml），输出R.java文件
aidl把.aidl文件转换为对应的java interface文件
javac编译所有.java文件，输出.class文件
dex工具把上面生成的.class文件转换为.dex文件
apkbuilder把所有没编译的资源（如图片），编译过的资源和dex文件打包输出为.apk文件
在release模式下，用zipalign工具对.apk进行对齐处理，以减少运行时内存占用

本文重点对第4步中.class经过dex到.dex过程源码进行分析。

0x02 android compile tasks分析

为了更好地分析.dex的产生过程，本文设定情景如下：

构建工具为gradle，采用android plugin 'com.android.application'，method数超过65535，需要进行multidex，并且指定了multiDexEnabled = true。

在shell终端cd到project根目录，输入：

gradle assemble

gradle进程会启动，在dex之前，进程控制流将进入VariantManager. createTasksForVariantData。添加完assemble task依赖后，会去调用taskManager.createTasksForVariantData(tasks, variantData)。由于android plugin为’com.android.application’，这里的taskManager是ApplicationTaskManager。

com/android/build/gradle/internal/VariantManager.java

/**
 * Create tasks for the specified variantData.
 */
public void createTasksForVariantData(
        final TaskFactory tasks,
        final BaseVariantData<? extends BaseVariantOutputData> variantData) {

    // Add dependency of assemble task on assemble build type task.
    tasks.named("assemble", new Action<Task>() {
        @Override
        public void execute(Task task) {
            BuildTypeData buildTypeData = buildTypes.get(
                            variantData.getVariantConfiguration().getBuildType().getName());
            task.dependsOn(buildTypeData.getAssembleTask());
        }
    });
    ...
        taskManager.createTasksForVariantData(tasks, variantData);
    }
}

ApplicationTaskManager.createTasksForVariantData()会通过ThreadRecorder.get().record()第二个callback参数的类型为Recorder.Block<Void>，在call回调中调用父类TaskManager.createPostCompilationTasks。ThreadRecorder可以记录该任务的在当前线程的执行时间，并且保证task之间是串行的。

/**
 * TaskManager for creating tasks in an Android application project.
 */
public class ApplicationTaskManager extends TaskManager {

    @Override
    public void createTasksForVariantData(
            @NonNull final TaskFactory tasks,
            @NonNull final BaseVariantData<? extends BaseVariantOutputData> variantData) {
            ...
        // Add a compile task
        ThreadRecorder.get().record(ExecutionType.APP_TASK_MANAGER_CREATE_COMPILE_TASK,
                new Recorder.Block<Void>() {
                    @Override
                    public Void call() {
                        AndroidTask<JavaCompile> javacTask = createJavacTask(tasks, variantScope);

                        if (variantData.getVariantConfiguration().getUseJack()) {
                            createJackTask(tasks, variantScope);
                        } else {
                            setJavaCompilerTask(javacTask, tasks, variantScope);
                            createJarTask(tasks, variantScope);
                            createPostCompilationTasks(tasks, variantScope);
                        }
                        return null;
                    }
                });
                ...
    }
}

TaskManager.createPostCompilationTasks方法，这个方法比较长，我们分段来分析。

首先从config得到isMultiDexEnabled，isMultiDexEnabled，isLegacyMultiDexMode，由于已经假设当前为需要MultiDex的场景，因此isMultiDexEnabled为true。若isMinifyEnabled也为true，则说明输入jar包需要进行混淆，本场景先不考虑。

TaskManager.java


/**
 * Creates the post-compilation tasks for the given Variant.
 *
 * These tasks create the dex file from the .class files, plus optional intermediary steps like
 * proguard and jacoco
 *
 */
public void createPostCompilationTasks(TaskFactory tasks, @NonNull final VariantScope variantScope) {
    checkNotNull(variantScope.getJavacTask());

    final ApkVariantData variantData = (ApkVariantData) variantScope.getVariantData();
    final GradleVariantConfiguration config = variantData.getVariantConfiguration();

    TransformManager transformManager = variantScope.getTransformManager();
...
    boolean isMinifyEnabled = config.isMinifyEnabled();
    boolean isMultiDexEnabled = config.isMultiDexEnabled();
    boolean isLegacyMultiDexMode = config.isLegacyMultiDexMode();

    AndroidConfig extension = variantScope.getGlobalScope().getExtension();

在支持MultiDex的场景中，先创建manifestKeepListTask，将依赖设置为ManifestProcessorTask，这些android compile task由AndroidTask<TransformTask>类型来描述。

接着创建multiDexClassListTask，依赖manifestKeepListTask。这两个tasks用来输出maindexlist.txt，其中包含了MainDex中必须的class，可参见MultiDex中出现的main dex capacity exceeded解决之道。


// ----- Multi-Dex support

AndroidTask<TransformTask> multiDexClassListTask = null;
// non Library test are running as native multi-dex
if (isMultiDexEnabled && isLegacyMultiDexMode) {
    if (AndroidGradleOptions.useNewShrinker(project)) {
        throw new IllegalStateException("New shrinker + multidex not supported yet.");
    }

    // ----------
    // create a transform to jar the inputs into a single jar.
    if (!isMinifyEnabled) {
        // merge the classes only, no need to package the resources since they are
        // not used during the computation.
        JarMergingTransform jarMergingTransform = new JarMergingTransform(
                TransformManager.SCOPE_FULL_PROJECT);
        transformManager.addTransform(tasks, variantScope, jarMergingTransform);
    }
    
    // ----------
    // Create a task to collect the list of manifest entry points which are
    // needed in the primary dex
    AndroidTask<CreateManifestKeepList> manifestKeepListTask = androidTasks.create(tasks,
            new CreateManifestKeepList.ConfigAction(variantScope));
    manifestKeepListTask.dependsOn(tasks,
            variantData.getOutputs().get(0).getScope().getManifestProcessorTask());

    // ---------
    // create the transform that's going to take the code and the proguard keep list
    // from above and compute the main class list.
    MultiDexTransform multiDexTransform = new MultiDexTransform(
            variantScope.getManifestKeepListFile(),
            variantScope,
            null);
    multiDexClassListTask = transformManager.addTransform(
            tasks, variantScope, multiDexTransform);
    multiDexClassListTask.dependsOn(tasks, manifestKeepListTask);
}

最后创建dexTask，这个用来把.class文件转为.dex的task，它依赖multiDexClassListTask。

    // create dex transform
    DexTransform dexTransform = new DexTransform(
            extension.getDexOptions(),
            config.getBuildType().isDebuggable(),
            isMultiDexEnabled,
            isMultiDexEnabled && isLegacyMultiDexMode ? variantScope.getMainDexListFile() : null,
            variantScope.getPreDexOutputDir(),
            variantScope.getGlobalScope().getAndroidBuilder(),
            getLogger());
    AndroidTask<TransformTask> dexTask = transformManager.addTransform(
            tasks, variantScope, dexTransform);
    // need to manually make dex task depend on MultiDexTransform since there's no stream
    // consumption making this automatic
    dexTask.optionalDependsOn(tasks, multiDexClassListTask);
}

task执行时，gradle引擎会去调用含有@TaskAction注解的方法，TransformTask类拥有Transfrom类型字段，其transform方法被标记为@TaskAction。同样通过ThreadRecorder.get().record中回调call()，执行transform.transform()

TransformTask.java

/**
 * A task running a transform.
 */
@ParallelizableTask
public class TransformTask extends StreamBasedTask implements Context {

    private Transform transform;
    ...
    @TaskAction
    void transform(final IncrementalTaskInputs incrementalTaskInputs)
            throws IOException, TransformException, InterruptedException {
		 ...
        ThreadRecorder.get().record(ExecutionType.TASK_TRANSFORM,
                new Recorder.Block<Void>() {
                    @Override
                    public Void call() throws Exception {
                        transform.transform(
                                TransformTask.this,
                                consumedInputs.getValue(),
                                referencedInputs.getValue(),
                                outputStream != null ? outputStream.asOutput() : null,
                                isIncremental.getValue());
                        return null;
                    }
                },
                new Recorder.Property("project", getProject().getName()),
                new Recorder.Property("transform", transform.getName()),
                new Recorder.Property("incremental", Boolean.toString(transform.isIncremental())));
    }

上述android compile tasks关系可以用下图描述：

从gradle task角度上看，这些task都属于TransformTask（继承至DefaultTask），它们区别仅在于transform字段。DexTask是本文主要关心的task，下面分析这个task执行过程中都做了什么。

0x03 DexTask执行过程分析

android build system中dex过程发生在DexTask，DexTask关联的Transform是DexTransform。

当DexTransform.transfrom方法被调用时，会先创建并初始化main目录作为输出dex的目录，然后调用androidBuilder.convertByteCode方法进行.class到.dex的转换，此时jarInputs为classes.jar，directoryInputs长度为空，传递的boolean类型的multiDex参数来自build.gralde文件中在defaultConfig对multiDexEnabled = true的设置。

DexTransform.java

@Override
public void transform(
        @NonNull Context context,
        @NonNull Collection<TransformInput> inputs,
        @NonNull Collection<TransformInput> referencedInputs,
        @Nullable TransformOutputProvider outputProvider,
        boolean isIncremental) throws TransformException, IOException, InterruptedException {
        ...
    // Gather a full list of all inputs.
    List<JarInput> jarInputs = Lists.newArrayList();
    List<DirectoryInput> directoryInputs = Lists.newArrayList();
    for (TransformInput input : inputs) {
        jarInputs.addAll(input.getJarInputs());
        directoryInputs.addAll(input.getDirectoryInputs());
    }
    
    
    try {
        // if only one scope or no per-scope dexing, just do a single pass that
        // runs dx on everything.
        if ((jarInputs.size() + directoryInputs.size()) == 1 || !dexOptions.getPreDexLibraries()) {
            File outputDir = outputProvider.getContentLocation("main",
                    getOutputTypes(), getScopes(),
                    Format.DIRECTORY);
            FileUtils.mkdirs(outputDir);

            // first delete the output folder where the final dex file(s) will be.
            FileUtils.emptyFolder(outputDir);

            // gather the inputs. This mode is always non incremental, so just
            // gather the top level folders/jars
            final List<File> inputFiles = Lists.newArrayList();
            for (JarInput jarInput : jarInputs) {
                inputFiles.add(jarInput.getFile());
            }

            for (DirectoryInput directoryInput : directoryInputs) {
                inputFiles.add(directoryInput.getFile());
            }

            androidBuilder.convertByteCode(
                    inputFiles,
                    outputDir,
                    multiDex,
                    mainDexListFile,
                    dexOptions,
                    null,
                    false,
                    true,
                    new LoggedProcessOutputHandler(logger));
        } else {

为了把输入的.class转换为.dex，AndroidBuilder.convertByteCode会另起进程去做dex，实际上是在新进程中exec dex工具，接下来我们进入dex源码，看看到底发生了什么。

 public void convertByteCode(
         @NonNull Collection<File> inputs,
         @NonNull File outDexFolder,
                  boolean multidex,
         @Nullable File mainDexList,
         @NonNull DexOptions dexOptions,
         @Nullable List<String> additionalParameters,
         boolean incremental,
         boolean optimize,
         @NonNull ProcessOutputHandler processOutputHandler)
         throws IOException, InterruptedException, ProcessException {
...
     BuildToolInfo buildToolInfo = mTargetInfo.getBuildTools();
     DexProcessBuilder builder = new DexProcessBuilder(outDexFolder);

     builder.setVerbose(mVerboseExec)
             .setIncremental(incremental)
             .setNoOptimize(!optimize)
             .setMultiDex(multidex)
             .setMainDexList(mainDexList)
             .addInputs(verifiedInputs.build());

     if (additionalParameters != null) {
         builder.additionalParameters(additionalParameters);
     }

     JavaProcessInfo javaProcessInfo = builder.build(buildToolInfo, dexOptions);

     ProcessResult result = mJavaProcessExecutor.execute(javaProcessInfo, processOutputHandler);
     result.rethrowFailure().assertNormalExitValue();
 }

0x04 dex过程分析

android 5.0中dex工具源码路径是dalvik/dx/src/com/android/dx，入口类是com.android.dx.command.Main，当解析到参数–dex时，转入com.android.dx.command.dexer.Main.main()

 public static void main(String[] args) {
...
     try {
	...
             if (arg.equals("--dex")) {
                 com.android.dx.command.dexer.Main.main(without(args, i));
                 break;
             } else if (arg.equals("--dump")) {
                 com.android.dx.command.dump.Main.main(without(args, i));
                 break;
             }
             ...
         }

main会调用com.android.dx.command.dexer.Main.run()，此时args.multiDex为true，直接进入runMultiDex

com.android.dx.command.dexer.Main.java

public static int run(Arguments arguments) throws IOException {
 ...
    try {
        if (args.multiDex) {
            return runMultiDex();
        } else {
            return runMonoDex();
        }
    } finally {
        closeOutput(humanOutRaw);
    }
}

runMultiDex会调用processAllFiles，第一行代码调用createDexFile()


 private static boolean processAllFiles() {
     createDexFile();
...

createDexFile先检查outputDex(: DexFile)字段是否为空，不为空则调用writeDex()把该dex的byte[]添加到dexOutputArrays(: List<byte[]>)。

writeDex()具体是通过outputDex.toDex(humanOutWriter, args.verboseDump)得到dex的byte[]。java中数组的下标是int类型，长度为32bits，因此一个dex文件最大理论是4G，但实际由于method, field数等限制，正常最大也就10M左右。

然后还会为outputDex字段新建一个DexFile对象，表示当前dex文件已经处理完毕，可以开始处理新的dex文件了。这里假设进程第一次执行createDexFile，因此outputDex为null。

private static void createDexFile() {
    if (outputDex != null) {
        dexOutputArrays.add(writeDex());
    }

    outputDex = new DexFile(args.dexOptions);

    if (args.dumpWidth != 0) {
        outputDex.setDumpWidth(args.dumpWidth);
    }
}

随后processAllFiles会根据args中numThreads来决定是否需要创建线程池。

if (args.numThreads > 1) {
    threadPool = Executors.newFixedThreadPool(args.numThreads);
    parallelProcessorFutures = new ArrayList<Future<Void>>();
}

接下来判断args.mainDexListFile，不为空说明指定了maindexlist.txt文件，这里假设不为空，filesNames数组是{‘path/way/to/classes.jar’}，长度为1。方法在for循环中调用processOne()

...
   anyFilesProcessed = false;
   String[] fileNames = args.fileNames;
   ...
   try {
       if (args.mainDexListFile != null) {
           // with --main-dex-list
           FileNameFilter mainPassFilter = args.strictNameCheck ? new MainDexListFilter() :
               new BestEffortMainDexListFilter();

           // forced in main dex
           for (int i = 0; i < fileNames.length; i++) {
               processOne(fileNames[i], mainPassFilter);
           }

processOne调用ClassPathOpener.process处理输入的classes.jar。ClassPathOpener会遍历classes.jar中的每个ZipEntry，读出byte[]，对每个ZipEntry在回调processFileBytes中调用Main.processFileBytes方法。

/**
 * Processes one pathname element.
 *
 * @param pathname {@code non-null;} the pathname to process. May
 * be the path of a class file, a jar file, or a directory
 * containing class files.
 * @param filter {@code non-null;} A filter for excluding files.
 */
private static void processOne(String pathname, FileNameFilter filter) {
    ClassPathOpener opener;

    opener = new ClassPathOpener(pathname, false, filter,
            new ClassPathOpener.Consumer() {

        @Override
        public boolean processFileBytes(String name, long lastModified, byte[] bytes) {
            return Main.processFileBytes(name, lastModified, bytes);
        }
...
   });

    if (args.numThreads > 1) {
        parallelProcessorFutures.add(threadPool.submit(new ParallelProcessor(opener)));
    } else {
        if (opener.process()) {
            anyFilesProcessed = true;
        }
    }
}

Main.processFileBytes把输入的bytes分为三类：

.class文件
.dex文件
资源文件

如果输入是.dex或资源文件，则把bytes分别写入libraryDexBuffers字段或outputResources字段，此时输入name(: String)为.class。当发现是class，则进一步调用processClass处理


 /**
  * Processes one file, which may be either a class or a resource.
  *
  * @param name {@code non-null;} name of the file
  * @param bytes {@code non-null;} contents of the file
  * @return whether processing was successful
  */
 private static boolean processFileBytes(String name, long lastModified, byte[] bytes) {
     boolean isClass = name.endsWith(".class");
     boolean isClassesDex = name.equals(DexFormat.DEX_IN_JAR_NAME);
     boolean keepResources = (outputResources != null);
... 
     String fixedName = fixPath(name);

     if (isClass) {

         if (keepResources && args.keepClassesInJar) {
             synchronized (outputResources) {
                 outputResources.put(fixedName, bytes);
             }
         }
         if (lastModified < minimumFileAge) {
             return true;
         }
         return processClass(fixedName, bytes);
     } else if (isClassesDex) {
         synchronized (libraryDexBuffers) {
             libraryDexBuffers.add(bytes);
         }
         return true;
     } else {
         synchronized (outputResources) {
             outputResources.put(fixedName, bytes);
         }
         return true;
     }
 }

processClass方法主要做了以下几件事：

为传入的class创建DirectClassFile对象，对应.class字节码文件
得到已经生成的dex的numMethodIds，numFieldIds
得到新Class的constantPoolSize，计算maxMethodIdsInDex = numMethodIds + constantPoolSize + 新Class的方法数 + 2个预留method，计算maxFieldIdsInDex = numFieldIds + constantPoolSize + 新Class的字段数 + 9个预留field
一旦发现maxMethodIdsInDex > args.maxNumberOfIdxPerDex 或者 maxFieldIdsInDex > args.maxNumber OfIdxPerDex，说明当前dex已经满了，调用createDexFile创建新dex来容纳该Class
否则，通过CfTranslator.translate方法将输入的DirectClassFile对象，得到ClassDefItem，添加到outputDex(: DexFile)

由此可以看出：

secondray dex中的class是根据classes.jar中ZipEntry的遍历顺序添加的。


/**
  * Processes one classfile.
  *
  * @param name {@code non-null;} name of the file, clipped such that it
  * <i>should</i> correspond to the name of the class it contains
  * @param bytes {@code non-null;} contents of the file
  * @return whether processing was successful
  */
 private static boolean processClass(String name, byte[] bytes) {
     if (! args.coreLibrary) {
         checkClassName(name);
     }

     DirectClassFile cf =
         new DirectClassFile(bytes, name, args.cfOptions.strictNameCheck);

     cf.setAttributeFactory(StdAttributeFactory.THE_ONE);
     cf.getMagic();

     int numMethodIds = outputDex.getMethodIds().items().size();
     int numFieldIds = outputDex.getFieldIds().items().size();
     int constantPoolSize = cf.getConstantPool().size();

     int maxMethodIdsInDex = numMethodIds + constantPoolSize + cf.getMethods().size() +
             MAX_METHOD_ADDED_DURING_DEX_CREATION;
     int maxFieldIdsInDex = numFieldIds + constantPoolSize + cf.getFields().size() +
             MAX_FIELD_ADDED_DURING_DEX_CREATION;

     if (args.multiDex
         // Never switch to the next dex if current dex is already empty
         && (outputDex.getClassDefs().items().size() > 0)
         && ((maxMethodIdsInDex > args.maxNumberOfIdxPerDex) ||
             (maxFieldIdsInDex > args.maxNumberOfIdxPerDex))) {
         DexFile completeDex = outputDex;
         createDexFile();
         assert  (completeDex.getMethodIds().items().size() <= numMethodIds +
                 MAX_METHOD_ADDED_DURING_DEX_CREATION) &&
                 (completeDex.getFieldIds().items().size() <= numFieldIds +
                 MAX_FIELD_ADDED_DURING_DEX_CREATION);
     }

     try {
         ClassDefItem clazz =
             CfTranslator.translate(cf, bytes, args.cfOptions, args.dexOptions, outputDex);
         synchronized (outputDex) {
             outputDex.add(clazz);
         }
         return true;

     } catch (ParseException ex) {
         DxConsole.err.println("\ntrouble processing:");
         if (args.debug) {
             ex.printStackTrace(DxConsole.err);
         } else {
             ex.printContext(DxConsole.err);
         }
     }
     errors.incrementAndGet();
     return false;
 }

再回到processAllFiles，前面假设指定了maindexlist，如果minialMainDex也为true的话，会立即创建新的DexFile，保证这个main dex中只包含maindexlist里的类，如何指定可以参考MultiDex中出现的main dex capacity exceeded解决之道 0x05。前面没有过滤掉的class都会放入到secondary dex。


        if (dexOutputArrays.size() > 0) {
            throw new DexException("Too many classes in " + Arguments.MAIN_DEX_LIST_OPTION
                    + ", main dex capacity exceeded");
        }

        if (args.minimalMainDex) {
            // start second pass directly in a secondary dex file.
            createDexFile();
        }

        // remaining files
        for (int i = 0; i < fileNames.length; i++) {
            processOne(fileNames[i], new NotFilter(mainPassFilter));
        }
    } else {
        // without --main-dex-list
        for (int i = 0; i < fileNames.length; i++) {
            processOne(fileNames[i], ClassPathOpener.acceptAll);
        }
    }
} catch (StopProcessing ex) {
    /*
     * Ignore it and just let the error reporting do
     * their things.
     */
}

在runMultiDex的最后，dex文件将以classes(..N).dex的形式输出在由args.outName指定的目录之下。

private static int runMultiDex() throws IOException {
		...
        } else if (args.outName != null) {
            File outDir = new File(args.outName);
            assert outDir.isDirectory();
            for (int i = 0; i < dexOutputArrays.size(); i++) {
                OutputStream out = new FileOutputStream(new File(outDir, getDexFileName(i)));
                try {
                    out.write(dexOutputArrays.get(i));
                } finally {
                    closeOutput(out);
                }
            }

        }

0x05 结论

通过对android build system中android plugin tasks和dx工具源码的分析，我们可以得出如下结论：

.dex文件本质上是.class文件经过com.android.dx.dex.file.DexFile.toDex方法转换得到
Secondary dex是在指定了multiDexEnabled = true且MainDex满足65535限制，或者指定multiDexEnabled = true和minimalMainDex = true的情况下，才会创建的dex，其包含的class是根据classes.jar中ZipEntry的遍历顺序添加的。