Change only updated files when using watch option.

[vilicvane]

Updating all files slows down debugging.

Compile on save would also do but I can't expect it would soon be there in Visual Studio Code, even it would be there, it might not be stable enough since the very beginning. It would be great if tsc updates changed files only when doing incremental compilation.

[DanielRosenwasser]

We already cache the bound ASTs during --watch to speed things up, but the problem with holding off on emit is that it's difficult to see what files might change in the process. Imagine two files, a.ts and b.ts.

a.ts

const enum E {
    Hello, World
}

b.ts

var x = [E.Hello, E.World]

When compiled, b.js will be

var x = [0, 1]

If we change a.ts to the following

const enum E {
    Hello, Beautiful, World
}

b.js is also affected and it's difficult to track that:

var x = [0, 2]

I could see us caching a hash of the emit result to avoid an IO penalty but I'd have to see some numbers indicating whether this becomes a real win.

---

Edit: I should clarify - when I say that it's difficult, I mean that it would introduce complexity in the compiler which I don't think is warranted.

[CyrusNajmabadi]

Are you trying to save the cost of compiling the file? Or just the cost of writing the file to disk?

We cannot avoid the former. We possibly could avoid the latter. Though we'd need numbers to know what sort of benefit there would be.

[vilicvane]

Thanks for your replies.

IO seems to be an issue on (my) Windows, though I can't figure out why (adding exception to Windows Defender makes it better, but still slow comparing to Mac). http://stackoverflow.com/questions/30046811/why-is-compiling-typescript-on-mac-much-faster-than-on-pc

On the other hand, many tools have "watch" functionality to detect file changes and do some re-releasing or restarting. I managed to make the part I can control smoother by determine compiling status, but I can't change third-party tools we use. So they are purely file changes + timeout based. Which means one more js file the compiler updated, one more js file that tool will need to re-releasing. Also, if two files are emitted not within time that's short enough, that tool might run more than once.

These kind of slow down our debugging process.

[CyrusNajmabadi]

Sounds like a reasonable suggestion. We could definitely attempt to avoid writes to files if we've emitted the exact same bytes as what exists on disk. We'd likely need to still read the files in to verify if they changed or not. But at least we wouldn't be writing (and updating timestamps) for anything unchanged.

[vilicvane]

That would definitely help, looking forward to it.

[scriby]

I noticed this as well recently, and ended up using grunt-ts, which will only pass changed files to the compiler and reduces output significantly

[vilicvane]

@scriby FYI, I made some changes to sys.ts, and replaced tsc.js file of my team as a temporary solution.

function compareFileWithData(fileName: string, data: string, writeByteOrderMark: boolean): boolean {
    if (!_fs.existsSync(fileName)) {
        return false;
    }
    var buffer = _fs.readFileSync(fileName);
    var len = buffer.length;
    if (len >= 2 && buffer[0] === 0xFE && buffer[1] === 0xFF) {
        // Big endian UTF-16 byte order mark detected. Since big endian is not supported by node.js,
        // flip all byte pairs and treat as little endian.
        return false;
    }
    if (len >= 2 && buffer[0] === 0xFF && buffer[1] === 0xFE) {
        // Little endian UTF-16 byte order mark detected
        return false;
    }
    if (len >= 3 && buffer[0] === 0xEF && buffer[1] === 0xBB && buffer[2] === 0xBF) {
        // UTF-8 byte order mark detected
        return writeByteOrderMark && buffer.toString("utf8", 3) == data;
    }
    // Default is UTF-8 with no byte order mark
    return !writeByteOrderMark && buffer.toString("utf8") == data;
}

function writeFile(fileName: string, data: string, writeByteOrderMark?: boolean): void {
    // If the file is exactly the same as what will be written, skip it.
    if (compareFileWithData(fileName, data, writeByteOrderMark)) {
        return;
    }

    // If a BOM is required, emit one
    if (writeByteOrderMark) {
        data = '\uFEFF' + data;
    }

    _fs.writeFileSync(fileName, data, "utf8");
}

[DanielRosenwasser]

@vilic can you describe the type of speedup you're getting from this? Is it substantial? What is the size of the codebase you are --watching?

[vilicvane]

Some numbers:

Modified

Initial compilation.

files:             329
lines:           64319
nodes:          219998
identifiers:     74564
symbols:         55576
types:           21180
memory used:   124673k
i/o read:        0.10s
i/o write:       0.12s
parse time:      0.67s
bind time:       0.36s
check time:      1.26s
emit time:       0.54s

Incremental compilation.

files:             329
lines:           64319
nodes:          219998
identifiers:     74564
symbols:         55576
types:           21180
memory used:   166245k
i/o read:        0.02s
i/o write:       0.14s
parse time:      0.06s
bind time:       0.01s
check time:      1.65s
emit time:       0.47s
total time:      2.19s

Unmodified

Initial compilation.

files:             329
lines:           64319
nodes:          219998
identifiers:     74564
symbols:         55576
types:           21180
memory used:   120182k
i/o read:        0.08s
i/o write:       0.49s
parse time:      0.64s
bind time:       0.35s
check time:      1.22s
emit time:       0.97s

Incremental compilation.

files:             329
lines:           64319
nodes:          219998
identifiers:     74564
symbols:         55576
types:           21180
memory used:   154028k
i/o read:        0.01s
i/o write:       0.64s
parse time:      0.05s
bind time:       0.00s
check time:      1.82s
emit time:       1.19s

These are numbers when third party tools are not watching, it seems that keep them watching will dramatically increase i/o write time (up to about 2 seconds, on (my) Windows).

And updating only necessary files speeds up re-releasing process quite a lot.

[CyrusNajmabadi]

So IO-write goes from around 0.6s to around 0.1 seconds. A reasonable savings, though not huge.

I'm curious what third party tools you're using that are watching this?

[vilicvane]

We are using yog, which is an integrated solution for fis on NodeJS. We are moving our project from PHP to NodeJS, before yog we were using another integrated solution for fis named fisp.

The integrated solution (yog/fisp) divided a project to several apps, and we need to watch and release for apps we are working on, separately.

[mhegazy]

We would consider a PR for this. things to watch for are the size of the change (a non-invasive/destabilizing would be ideal), also a quick way to verify that the file has not changed would be ideal.