benchmarking: measure and analyze scripts

[sophiebits]

This uses wall-clock time (for now) so it's noisier than alternatives (cachegrind, CPU perf-counters), but it's still valuable. In a future diff we can make it use those.

measure.py outputs something that analyze.py can understand, but you can use analyze.py without measure.py too. The file format is simple:

$ cat measurements.txt
factory_ms_jsc_jit 13.580322265625
factory_ms_jsc_jit 13.659912109375
factory_ms_jsc_jit 13.67919921875
factory_ms_jsc_nojit 12.827880859375
factory_ms_jsc_nojit 13.105224609375
factory_ms_jsc_nojit 13.195068359375
factory_ms_node 40.4891400039196
factory_ms_node 40.6669420003891
factory_ms_node 43.52413299679756
ssr_pe_cold_ms_jsc_jit 43.06005859375
...

(The lines do not need to be sorted.)

Comparing 0.14.0 vs master:

$ ./measure.py react-0.14.0.min.js >014.txt
Measuring SSR for PE benchmark (30 trials)
..............................
Measuring SSR for PE with warm JIT (3 slow trials)
...
$ ./measure.py react.min.js >master.txt
Measuring SSR for PE benchmark (30 trials)
..............................
Measuring SSR for PE with warm JIT (3 slow trials)
...
$ ./analyze.py 014.txt master.txt
Comparing 014.txt (control) vs master.txt (test)
Significant differences marked by ***
% change from control to test, with 99% CIs:

* factory_ms_jsc_jit
    % change:  -0.56% [ -2.51%,  +1.39%]
    means: 14.037 (control), 13.9593 (test)
* factory_ms_jsc_nojit
    % change:  +1.23% [ -1.18%,  +3.64%]
    means: 13.2586 (control), 13.4223 (test)
* factory_ms_node
    % change:  +3.53% [ +0.29%,  +6.77%]  ***
    means: 42.0529 (control), 43.54 (test)
* ssr_pe_cold_ms_jsc_jit
    % change:  -6.84% [ -9.04%,  -4.65%]  ***
    means: 44.2444 (control), 41.2187 (test)
* ssr_pe_cold_ms_jsc_nojit
    % change: -11.81% [-14.66%,  -8.96%]  ***
    means: 52.9449 (control), 46.6953 (test)
* ssr_pe_cold_ms_node
    % change:  -2.70% [ -4.52%,  -0.88%]  ***
    means: 96.8909 (control), 94.2741 (test)
* ssr_pe_warm_ms_jsc_jit
    % change: -17.60% [-22.04%, -13.16%]  ***
    means: 13.763 (control), 11.3439 (test)
* ssr_pe_warm_ms_jsc_nojit
    % change: -20.65% [-22.62%, -18.68%]  ***
    means: 30.8829 (control), 24.5074 (test)
* ssr_pe_warm_ms_node
    % change:  -8.76% [-13.48%,  -4.03%]  ***
    means: 30.0193 (control), 27.3964 (test)
$

[sophiebits]

Sorry for Python but numpy/scipy seem way better and more comprehensive than anything available in JS.

[sophiebits]

Preview of what the numbers look like with instruction counters instead of wall-clock time:

$ python analyze.py 014.txt master.txt
Comparing 014.txt (control) vs master.txt (test)
Significant differences marked by ***
% change from control to test, with 99% CIs:

* factory_instr_jsc_jit
    % change:  +0.90% [ +0.83%,  +0.97%]  ***
    means: 7.83409e+07 (control), 7.9047e+07 (test)
* factory_instr_jsc_nojit
    % change:  +0.78% [ +0.69%,  +0.88%]  ***
    means: 7.58159e+07 (control), 7.64087e+07 (test)
* factory_instr_node
    % change:  +1.06% [ +1.00%,  +1.11%]  ***
    means: 1.93569e+08 (control), 1.95612e+08 (test)
* ssr_pe_cold_instr_jsc_jit
    % change: -13.82% [-13.95%, -13.70%]  ***
    means: 2.30181e+08 (control), 1.98365e+08 (test)
* ssr_pe_cold_instr_jsc_nojit
    % change: -21.54% [-21.65%, -21.43%]  ***
    means: 3.02415e+08 (control), 2.37276e+08 (test)
* ssr_pe_cold_instr_node
    % change:  -4.50% [ -4.67%,  -4.32%]  ***
    means: 3.30435e+08 (control), 3.15576e+08 (test)
* ssr_pe_warm_instr_jsc_jit
    % change: -30.84% [-32.14%, -29.55%]  ***
    means: 9.6651e+07 (control), 6.68413e+07 (test)
* ssr_pe_warm_instr_jsc_nojit
    % change: -32.21% [-32.58%, -31.84%]  ***
    means: 1.97639e+08 (control), 1.3398e+08 (test)
* ssr_pe_warm_instr_node
    % change:  -6.55% [-10.94%,  -2.16%]  ***
    means: 1.2915e+08 (control), 1.20714e+08 (test)

The improvements are more precise but more exaggerated. I am not yet sure why. Maybe including memory accesses will make a difference.