Memory leak and performance issue

[CorvusCorrax]

Generating large file (> 10000 pages) crash node on my system (Linux Fedora 23) :
JavaScript heap out of memory

Here is my test script :

var PDFDocument = require('pdfkit');
var fs = require('fs');
var doc = new PDFDocument();
var v8 = require('v8');
var pageCount = 20000;

var getStats = function(text) {
  var stats = v8.getHeapSpaceStatistics();
  stats.forEach(function(stat) {
    console.log(text + ' ' + stat.space_name + ' available size : ' + stat.space_available_size);
  });
};

doc.pipe(fs.createWriteStream('test.pdf'));

console.time('Pdf generation time');
doc.on('end', function() {
  console.log();
  getStats('Final memory');
  console.log();
  console.timeEnd('Pdf generation time');
});

console.log('Start generating ' + pageCount + ' pages');
getStats('Starting memory');

for (var i = 1; i < pageCount; ++i) {
  for (var y = 0; y < 40; ++y) {
    doc.text('testText', y, y);
  }
  doc.addPage();
  if (i % 4000 === 0) {
    console.log();
    getStats('Page ' + i);
  }
}
doc.end();

And here is the result :

Start generating 20000 pages
Starting memory new_space available size : 775352
Starting memory old_space available size : 122488
Starting memory code_space available size : 231872
Starting memory map_space available size : 450104
Starting memory large_object_space available size : 1472650752

Page 4000 new_space available size : 8944344
Page 4000 old_space available size : 33672632
Page 4000 code_space available size : 1745920
Page 4000 map_space available size : 563520
Page 4000 large_object_space available size : 958758400

Page 8000 new_space available size : 9308184
Page 8000 old_space available size : 65610464
Page 8000 code_space available size : 1249696
Page 8000 map_space available size : 563432
Page 8000 large_object_space available size : 461798912

Page 12000 new_space available size : 1275960
Page 12000 old_space available size : 62591016
Page 12000 code_space available size : 782144
Page 12000 map_space available size : 564048
Page 12000 large_object_space available size : 0

<--- Last few GCs --->

[30138:0x34205a0]   218726 ms: Mark-sweep 1400.9 (1545.8) -> 1400.9 (1541.3) MB, 2200.4 / 0.1 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2201 ms) last resort 
[30138:0x34205a0]   221096 ms: Mark-sweep 1400.9 (1541.3) -> 1400.9 (1541.3) MB, 2368.5 / 0.1 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2313edd29891 <JS Object>
    2: _text [/home/dev/svn/testPdfKit/node_modules/pdfkit/js/mixins/text.js:~33] [pc=0x9b80fbc8006](this=0x3c6fdb4a5a99 <a PDFDocument with map 0x1bebc6d107f9>,text=0x19e7d0cf6569 <String[8]: testText>,x=29,y=29,options=0x38aeea002311 <undefined>,lineCallback=0x69f67fdc639 <JS BoundFunction (BoundTargetFunction 0x3c6fdb4a9e31)>)
    3: /* anonymous */ [/home/dev/svn/testPdfKit/pdfTest.js:~1] [...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x13647ec [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewByteArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::SourcePositionTableBuilder::ToSourcePositionTable(v8::internal::Isolate*, v8::internal::Handle<v8::internal::AbstractCode>) [node]
 7: v8::internal::FullCodeGenerator::MakeCode(v8::internal::CompilationInfo*, unsigned long) [node]
 8: v8::internal::FullCodegenCompilationJob::ExecuteJobImpl() [node]
 9: v8::internal::CompilationJob::ExecuteJob() [node]
10: 0xd97d90 [node]
11: 0xd98b28 [node]
12: 0xd98d8f [node]
13: 0xd9ce59 [node]
14: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [node]
15: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [node]
16: 0x9b80f68437d
Abandon (core dumped)

After some analysis, i found out that the writable streams from referencePDF object eats a lot of memory.
Same test result after removing the streams :

Start generating 20000 pages
Starting memory new_space available size : 2121376
Starting memory old_space available size : 600
Starting memory code_space available size : 352
Starting memory map_space available size : 450104
Starting memory large_object_space available size : 1472650752

Page 4000 new_space available size : 2915960
Page 4000 old_space available size : 21162392
Page 4000 code_space available size : 322848
Page 4000 map_space available size : 562200
Page 4000 large_object_space available size : 1162141184

Page 8000 new_space available size : 7767744
Page 8000 old_space available size : 69523024
Page 8000 code_space available size : 0
Page 8000 map_space available size : 562200
Page 8000 large_object_space available size : 877977088

Page 12000 new_space available size : 9225224
Page 12000 old_space available size : 101655856
Page 12000 code_space available size : 0
Page 12000 map_space available size : 562376
Page 12000 large_object_space available size : 589618688

Page 16000 new_space available size : 5595376
Page 16000 old_space available size : 133766616
Page 16000 code_space available size : 0
Page 16000 map_space available size : 562200
Page 16000 large_object_space available size : 301784576

Final memory new_space available size : 4856536
Final memory old_space available size : 569029752
Final memory code_space available size : 154112
Final memory map_space available size : 566336
Final memory large_object_space available size : 22240768

Pdf generation time: 82549.040ms

It seems that large_object_space available memory gets eaten up a lot more slowly and the process also goes faster :
2000 pages with streams :

Start generating 2000 pages
Starting memory new_space available size : 775320
Starting memory old_space available size : 122472
Starting memory code_space available size : 234816
Starting memory map_space available size : 450104
Starting memory large_object_space available size : 1472650752

Final memory new_space available size : 8711240
Final memory old_space available size : 13663000
Final memory code_space available size : 285504
Final memory map_space available size : 561408
Final memory large_object_space available size : 1191419392

Pdf generation time: 5894.804ms

2000 pages without streams :

Start generating 2000 pages
Starting memory new_space available size : 9768096
Starting memory old_space available size : 428104
Starting memory code_space available size : 185024
Starting memory map_space available size : 152
Starting memory large_object_space available size : 1427037696

Final memory new_space available size : 13427376
Final memory old_space available size : 14697760
Final memory code_space available size : 1355936
Final memory map_space available size : 373800
Final memory large_object_space available size : 1289543168

Pdf generation time: 4895.007ms

[CorvusCorrax]

Last commit definitely fix memory leak and boost performance (around x2 faster).
Here are the same tests as before with the new code :

Start generating 2000 pages
Starting memory new_space available size : 7224264
Starting memory old_space available size : 322656
Starting memory code_space available size : 201920
Starting memory map_space available size : 152
Starting memory large_object_space available size : 1427037696

Final memory new_space available size : 3823568
Final memory old_space available size : 632
Final memory code_space available size : 1674304
Final memory map_space available size : 458448
Final memory large_object_space available size : 1419042304

Pdf generation time: 3140.572ms

For 2000 pages, it goes from 5894.804ms to 3140.572ms.

Start generating 20000 pages
Starting memory new_space available size : 13445824
Starting memory old_space available size : 405848
Starting memory code_space available size : 254112
Starting memory map_space available size : 152
Starting memory large_object_space available size : 1427561984

Page 4000 new_space available size : 11201432
Page 4000 old_space available size : 20305960
Page 4000 code_space available size : 1793984
Page 4000 map_space available size : 72
Page 4000 large_object_space available size : 1397546496

Page 8000 new_space available size : 3306648
Page 8000 old_space available size : 15435752
Page 8000 code_space available size : 1217216
Page 8000 map_space available size : 977904
Page 8000 large_object_space available size : 1398595072

Page 12000 new_space available size : 6359464
Page 12000 old_space available size : 4751632
Page 12000 code_space available size : 1247104
Page 12000 map_space available size : 977904
Page 12000 large_object_space available size : 1393876480

Page 16000 new_space available size : 5812632
Page 16000 old_space available size : 8107840
Page 16000 code_space available size : 0
Page 16000 map_space available size : 977464
Page 16000 large_object_space available size : 1374477824

Page 20000 new_space available size : 4377208
Page 20000 old_space available size : 7403064
Page 20000 code_space available size : 0
Page 20000 map_space available size : 977904
Page 20000 large_object_space available size : 1362845184

Final memory new_space available size : 7157552
Final memory old_space available size : 13000616
Final memory code_space available size : 1209856
Final memory map_space available size : 973064
Final memory large_object_space available size : 1341873664

Pdf generation time: 33538.650ms

For 20000 pages, it goes from a JavaScript heap out of memory error (or 82549.040ms with previous fix) to 33538.650ms.

[rogierslag]

Any plans on merging this? The memory leak also happens when generating a large number of smaller PDF files, causing continuous service restarts

[fredlemieux]

Will there be a fix for the memory leak issues?

[rsshilli]

I brought down this branch last night (well, @blikblum's branch actually, where he has this merged in) and tried it out on a 1.4k page doc that I'm struggling to keep under 10GB. It wasn't that much different, maybe 8 GB only.

What I don't understand is that the final document is < 100 MB in size. Why does it take even 1 GB to create a 100 MB file, nevermind 10GB?

[rsshilli]

OK. I figured it out. It's a memory leak when adding too many images (my doc has a screenshot png on every page). I'll open up a new issue and submit another PR.

[blikblum]

Should be fixed on latest relese. Please reopen with an example if still getting errors