zhixingheyi_tian的博客

include/orc/Reader.hh /** * Options for creating a RowReader. */ class RowReaderOptions { private: ORC_UNIQUE_PTR<RowReaderOptionsPrivate> privateBits;src/Options.hh/** * RowReaderOptions Implementation */ struct RowReaderOpti

orc/c++/include/orc/Reader.hhRowReader/** * The interface for reading rows in ORC files. * This is an an abstract class that will be subclassed as necessary. */ class RowReader { public: virtual ~RowReader(); /** * Get the selec

ORC StreamOrc 在读取一个 stripe时，是安装stream为单位读取的，stripe 中的column可能只有一个stream，或者多个不同属性的stream组成，stream 不是 column的子单元，enum Kind { // boolean stream of whether the next value is non-null PRESENT = 0;...

RecordReaderImpl.java@Override public boolean nextBatch(VectorizedRowBatch batch) throws IOException { try { if (rowInStripe &amp;amp;gt;= rowCountInStripe) { currentStripe += 1; ...

ReaderImpl.java protected OrcTail tail;OrcTail.java public OrcProto.Footer getFooter() { return fileTail.getFooter(); }

不管是 readAllDataStreams 还是 readPartialDataStreams ，最后都要 经过 createStreamscreateStreamscreateStreams 最后的输出结果是这个Map<StreamName, InStream> streams每个 StreamName 对应的内存流void createStreams(List<O...

readPartialDataStreamsRecordReaderImpl.java private void readPartialDataStreams(StripeInformation stripe) throws IOException { List<OrcProto.Stream> streamList = stripeFooter.getStreamsLi...

readAllDataStreamsRecordReaderImpl.java private void readAllDataStreams(StripeInformation stripe) throws IOException { long start = stripe.getIndexLength(); long end = start + stripe.getDat...

ORC provides three level of indexes within each file:file level - statistics about the values in each column across the entire filestripe level - statistics about the values in each column for each ...

OrcColumnarBatchReader.java /** * Return true if there exists more data in the next batch. If exists, prepare the next batch * by copying from ORC VectorizedRowBatch columns to Spark ColumnarBa...