[Feat]: incremental file reader

[ShaMan123]

What is your idea?

I would like to propose an incremental file reader API that enables reading a file without loading all of it to memory.

API

type ParsedLine = ReturnType<ifcApi['getLine']>

readLine(raw:string): ParsedLine

readIfcFileIncrementally<T extends boolean = false>(file: string,  options: {
    onHeader?: (raw: string[]) => void;
    onLine?: (id: number, line: T extends true ? string : ParsedLine) => void;
    onEnd?: (header: string[]) => void;
    raw?: T,
    flattern?: boolean,
    inverse?: boolean,
  } = {}): Promise<void>

Challanges

Supporting flatten/inverse is the real challenge of the reader.
Considering the use case of an incremental reader it might not be needed.

There are a few approaches we can explore in order to correctly support flatten and inverse.

I can think of the following:

• First read: index dependencies
• Establish dependency reading order
• Second read: parse lines according to reading order

Workaround

I currently do something similar since I am using fragments split feature and need more context beyond what I get when splitting the file, e.g. indexing the file, see ThatOpen/engine_fragment#164.

Since I do not want to handle ifc parsing I workaround as follows:

  const api = new IfcAPI();
  await api.Init();

  let header: string[] = [];

  const readIFCLine = (id: number, raw: string) => {
    const br = "\r\n";
    const footer = "ENDSEC;" + br + br + "END-" + header[0];
    const ifcData = [header.join(br), raw, footer].join(br);
    const modelId = api.OpenModel(new TextEncoder().encode(ifcData), {
      COORDINATE_TO_ORIGIN: false,
    });
    const result = api.GetLine(modelId, id);
    api.CloseModel(modelId);
    return result;
  };

  await readIfcFileIncrementally(filePath, {
     raw: true,

    onHeader: (raw) => {
      header = raw;
    },

    onLine: (id, raw) => {
      const data = readIFCLine(id, raw);
    },
  });

[beachtom]

So this is a difficult problem - and not easily solveable I think in the general case - because we cannot assume IFC lines are grouped together. The parser has to jump around the file in many cases.

What is challenge here - is it memory usage or speed?

[ShaMan123]

memory