[2.8.0-rc] Segfault when running compiler with --d or --watch (out of memory)

[canonic-epicure]

Hello,

I was using 2.8.0-dev-20180216 and recently switched to 2.8.0-rc. I immediately noticed, that, when launching compiler with --watch it started to take a lot longer for the initial compilation. The subsequent compilations on file change were working fine.

But now, after some code refactoring, launching with --watch segfaults with out of memory error (after a long wait). Running w/o --watch works just fine (and fast enough).

So there's definitely some regression related to the initial compilation of the project in --watch mode.

Here's an output from the shell session. Note, how compilation w/o --watch worked fine (3rd command):

nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$ ./node_modules/.bin/tsc --version
Version 2.8.0-rc
nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$ node --version
v8.9.4
nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$ ./node_modules/.bin/tsc
nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$ ./node_modules/.bin/tsc --watch

16:40:41 - Starting compilation in watch mode...



<--- Last few GCs --->

[4089:0x23b2e20]    92678 ms: Mark-sweep 1411.0 (1459.8) -> 1411.0 (1443.8) MB, 1653.6 / 0.5 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1655 ms) last resort GC in old space requested
[4089:0x23b2e20]    94301 ms: Mark-sweep 1411.0 (1443.8) -> 1411.0 (1443.8) MB, 1622.7 / 0.6 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x43f5ef25ee1 <JSObject>
    1: getTextOfNode(aka getTextOfNode) [/home/nickolay/workspace/Bryntum/SchedulingEngine0.x/node_modules/typescript/lib/tsc.js:~60023] [pc=0xc63f1afcb97](this=0x212ba7e02311 <undefined>,node=0x27070b5c3fd9 <Node map = 0x136dd41e54c1>,includeTrivia=0x212ba7e02421 <false>)
    2: emitIdentifier(aka emitIdentifier) [/home/nickolay/workspace/Bryntum/SchedulingEngine0.x/node_modules/typescript/lib/t...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x122563c [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::LoadIC::LoadFromPrototype(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Smi>) [node]
 7: v8::internal::LoadIC::GetMapIndependentHandler(v8::internal::LookupIterator*) [node]
 8: v8::internal::IC::ComputeHandler(v8::internal::LookupIterator*) [node]
 9: v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [node]
10: v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>) [node]
11: v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0xc63f178463d
Aborted (core dumped)
nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$

[canonic-epicure]

Correction - it actually fails with 2.8.0-dev.20180216 as well, but now, it just takes forever for the initial compilation with --watch. Compilation w/o --watch works fine with 2.8.0-dev.20180216 as well.

[canonic-epicure]

The application does a heavy use of mixins if that matters.

[DanielRosenwasser]

Potentially a duplicate of #21140, but @sheetalkamat can you take a look?

[sheetalkamat]

@samuraijack can you please share your project so i can repro and investigate this. So far we haven't gotten repro on this issue so its hard to predict without looking at it what might be going wrong. Thanks.

[canonic-epicure]

@sheetalkamat Ok, I'll prepare the reproducible test case, on Monday already..

[canonic-epicure]

Unfortunately, preparing a test case is not easy - I can't share the code.

But you should check the code, that performs the initial compilation during --watch. That initial compilation takes a lot longer than normal compilation without --watch. I guess it also uses more memory and thus crash at certain project size. So that initial compilation is done in some different mode and bug is there.

[taoqf]

I've seen this too.
I believe this may have relation with #22668 if you use cnpm too. @samuraijack

[canonic-epicure]

Nope, we don't use cnpm. I believe its some algorithm with quadratic behavior in "watch" compilation mode. At certain project size it becomes noticeable and process crashes with OOM.

@sheetalkamat To reproduce this, you could try to generate huge dummy project, based on the template like:

type Constructable<T = {}> = new (...args : any[]) => T

export const Mixin1 = <T extends Constructable>(base : T) =>

    class Mixin1 extends base {
        method1() {
        }
    }

export const Mixin2 = <T extends Constructable>(base : T) =>

    class Mixin2 extends base {
        method2 () {
        }
    }


export const Mixin3 = <T extends Constructable>(base : T) =>

    class Mixin3 extends Mixin2<Constructable>(base) {
        method3 () {
        }
    }


export const Mixin4 = <T extends Constructable>(base : T) =>

    class Mixin4 extends Mixin1(Mixin3(base)) {
        method4 () {
            this.method1()

            this.method2()
        }
    }

The idea is that Mixin(N) includes Mixin(N-1). Mixins should be in different files. This roughly corresponds to what we have in our project (of course we don't have such deep mixins "inheritance", only 2-3 levels, but there are many sibling mixins). I believe at certain N the issue will manifest itself.

[sheetalkamat]

@samuraijack can you try the nightly build that would be published tonight and see if it still repros for you. I have been playing around with various number of files with the contents from the files you shared and it seem to be comparable:
Watch Mode:

c:\temp\test>node c:\TypeScript\built\local\tsc.js --extendedDiagnostics -w
15:59:34 - Starting compilation in watch mode...


Files:                 2001
Lines:                92523
Nodes:               353922
Identifiers:         118900
Symbols:             162671
Types:               180859
Memory used:        346460K
I/O Read time:        0.64s
Parse time:           0.69s
Program time:         1.50s
Bind time:            0.31s
Check time:           5.59s
transformTime time:   4.06s
commentTime time:     0.29s
I/O Write time:       2.15s
printTime time:       8.62s
Emit time:            8.72s
Total time:          16.12s
15:59:50 - Compilation complete. Watching for file changes.


16:00:20 - File change detected. Starting incremental compilation...


Files:                 2001
Lines:                92523
Nodes:               353922
Identifiers:         118900
Symbols:             100351
Types:                  146
Memory used:        347119K
I/O Read time:        0.00s
Parse time:           0.00s
Program time:         0.06s
Bind time:            0.04s
Check time:           0.14s
transformTime time:   0.10s
commentTime time:     0.00s
printTime time:       0.16s
Emit time:            0.17s
I/O Write time:       0.00s
Total time:           0.41s
16:00:21 - Compilation complete. Watching for file changes.


16:00:30 - File change detected. Starting incremental compilation...


Files:                 2001
Lines:                92523
Nodes:               353922
Identifiers:         118900
Symbols:             100351
Types:                  146
Memory used:        348419K
I/O Read time:        0.00s
Parse time:           0.01s
Program time:         0.05s
Bind time:            0.01s
Check time:           0.00s
transformTime time:   0.01s
commentTime time:     0.00s
printTime time:       0.01s
Emit time:            0.01s
I/O Write time:       0.00s
Total time:           0.07s
16:00:30 - Compilation complete. Watching for file changes.

Without watch:

c:\temp\test>node c:\TypeScript\built\local\tsc.js --extendedDiagnostics
Files:                 2001
Lines:                92523
Nodes:               353922
Identifiers:         118900
Symbols:             162671
Types:               160859
Memory used:        438993K
I/O Read time:        0.34s
Parse time:           0.85s
Program time:         1.29s
Bind time:            0.43s
Check time:           4.37s
transformTime time:   2.68s
commentTime time:     0.36s
I/O Write time:       2.82s
printTime time:       8.30s
Emit time:            8.30s
Total time:          14.39s

I got OOM for 20000 files with and without --watch option and works for both modes with 5000 files.

[canonic-epicure]

Tried with 2.9.0-dev.20180329 - no changes, did you mean 2.9.0-dev.20180330 perhaps?

I have only 54 TS files. The internal typing references are a lot more complex than in the example I posted of course..

[canonic-epicure]

Tried with 2.9.0-dev.20180330 - no change.