test: add test-benchmark-napi by forivall · Pull Request #23585 · nodejs/node

forivall · 2018-10-12T18:26:32Z

also makes sure that the napi benchmark is built before running jstest

skipped on windows since n-api benchmarks aren't built there yet

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

also makes sure that the napi benchmark is built before running jstest skipped on windows since n-api benchmarks aren't built there yet

addaleax · 2018-10-12T20:29:38Z

CI: https://ci.nodejs.org/job/node-test-pull-request/17764/

BridgeAR · 2018-10-12T21:08:58Z

test/sequential/test-benchmark-napi.js

+const common = require('../common');
+
+if (common.isWindows) {
+  common.skip('vcbuild.bat doesn\'t build the n-api benchmarks yet')


The linter fails on this due to the missing semicolon at the end.

Trott · 2018-10-13T00:09:50Z

You can run make lint-js to lint the JS files.

~~The new test is failing because it is running too many benchmark variants. I could explain in a GitHub comment, but we're in the same room right now, so I'll walk over....~~ Whoops, the test is failing because the Makefile task is probably not being run in CI...

Trott · 2018-10-13T03:21:44Z

CI: https://ci.nodejs.org/job/node-test-pull-request/17795

gireeshpunathil · 2018-10-13T04:32:08Z

open question: In other benchmark triggering tests we have seen checking for and skipping based on available memory - is this test an exemption?

forivall · 2018-10-13T16:41:05Z

@gireeshpunathil I noticed that the memcheck wasn't in all of the benchmark tests -- only the http and tls ones. buffer, child-process, dgram, net and path all do not have the memory restriction.
my intuition is that n-api was closest to buffer or child process

Trott · 2018-10-13T16:55:50Z

CI: https://ci.nodejs.org/job/node-test-pull-request/17813/

Trott · 2018-10-14T03:40:27Z

Resume Build CI: https://ci.nodejs.org/job/node-test-pull-request/17848/

richardlau · 2018-10-14T04:41:27Z

Failures are related (at least one of the n-api benchmark addons is failing to compile):
https://ci.nodejs.org/job/node-test-commit-linux/nodes=debian8-64/22354/console

23:43:23 ../napi_binding.c: In function ‘CallWithArray’:
23:43:23 ../napi_binding.c:53:5: error: ‘for’ loop initial declarations are only allowed in C99 or C11 mode
23:43:23      for (uint32_t i = 0; i < length; ++i) {
23:43:23      ^
23:43:23 ../napi_binding.c:53:5: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code
23:43:23 ../napi_binding.c: In function ‘CallWithArguments’:
23:43:23 ../napi_binding.c:176:5: error: ‘for’ loop initial declarations are only allowed in C99 or C11 mode
23:43:23      for (uint32_t i = 1; i < loop; ++i) {
23:43:23      ^
23:43:23 napi_binding.target.mk:99: recipe for target 'Release/obj.target/napi_binding/napi_binding.o' failed
23:43:23 make[2]: *** [Release/obj.target/napi_binding/napi_binding.o] Error 1
23:43:23 make[2]: *** Waiting for unfinished jobs....

We do set -std=gnu++1y for 'cflags_cc' in

node/common.gypi

Line 373 in f569ecf

'cflags_cc': [ '-fno-rtti', '-fno-exceptions', '-std=gnu++1y' ],

but I think this only applies to .cc files (napi_binding.c is a .c).

cc @nodejs/addon-api @nodejs/gyp

richardlau · 2018-10-14T04:42:46Z

Sorry, meant to cc @nodejs/n-api instead of addon-api.

NickNaso · 2018-10-14T08:32:14Z

Hi @richardlau,
maybe adding the following option "-std=c11" to the CFLAGS could solve the problem.
Now the options for CFLAGS are:

'cflags': [ '-Wall', '-Wextra', '-Wno-unused-parameter', ]

change in

'cflags': [ '-Wall', '-Wextra', '-Wno-unused-parameter', '-std=c11' ]

See:

node/common.gypi

Line 372 in f569ecf

'cflags': [ '-Wall', '-Wextra', '-Wno-unused-parameter', ],

Trott · 2018-10-17T04:40:14Z

~~Added @NickNaso's suggestion.~~

~~CI: https://ci.nodejs.org/job/node-test-pull-request/17928/~~

richardlau · 2018-10-17T04:43:17Z

common.gypi

Since common.gypi is used by addons we should at least run this past citgm.

I suppose an alternative is to hard-code the flag into the Makefile for the napi benchmark build step only?

Or maybe in the binding.gyp of the n-api benchmark addon?

No need for CITGM. It's failing to build on lots of hosts. Will force-push that commit out....

Trott · 2018-10-17T04:58:43Z

Rewrote the loops to declare the variables before using in for.

CI: https://ci.nodejs.org/job/node-test-pull-request/17930/

Trott · 2018-10-19T04:22:26Z

Added common.skip() for workers.

CI: https://ci.nodejs.org/job/node-test-pull-request/17982/

Trott · 2018-10-20T01:41:36Z

For ubuntu1604_sharedlibs_debug_x64, do we need to skip the test or is there a bug to be fixed?

https://ci.nodejs.org/job/node-test-commit-linux-containered/7955/nodes=ubuntu1604_sharedlibs_debug_x64/console

22:29:34 not ok 2090 sequential/test-benchmark-napi
22:29:34   ---
22:29:34   duration_ms: 15.590
22:29:34   severity: fail
22:29:34   exitcode: 1
22:29:34   stack: |-
22:29:34     /home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_debug_x64/benchmark/napi/function_args/index.js: V8 Binding failed to load
22:29:34     misc/function_call.js Binding failed to load
22:29:34     assert.js:351
22:29:34         throw err;
22:29:34         ^
22:29:34     
22:29:34     AssertionError [ERR_ASSERTION]: benchmark file not running exactly one configuration in test: 
22:29:34     napi/function_args
22:29:34     
22:29:34     napi/function_call
22:29:34     
22:29:34         at ChildProcess.child.on (/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_debug_x64/test/common/benchmark.js:43:12)
22:29:34         at ChildProcess.emit (events.js:182:13)
22:29:34         at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
22:29:34   ...

richardlau · 2018-10-20T02:17:00Z

The job config for ubuntu1604_sharedlibs_debug_x64 only runs build-ci -- it manually invokes test.py rather than any of the test-* targets (the other nodes in the job run run-ci). It therefore does not exercise the makefile changes in this PR and does not build the addons required to run the benchmark.

There may also be an issue in the benchmark as it is requiring from build/Release which is probably not what we want in a debug build (assuming the addon was built in the first place).

cc @nodejs/build

refack · 2018-10-20T02:25:27Z

'cflags': [ '-Wall', '-Wextra', '-Wno-unused-parameter', '-std=c11' ]

This could be added to

node/benchmark/napi/function_args/binding.gyp

Lines 3 to 6 in dd64946

    
           { 
        
             'target_name': 'napi_binding', 
        
             'sources': [ 'napi_binding.c' ] 
        
           },

I think we'll need to move the test to /test/addons-napi, otherwise we'll need to add more singleton exception to te CI config... (Also that will eliminate singleton make targets)

Ref: #23585 (comment) Python's `list.remove` will throw if the element is not found and also it removes only the first occurrence. This patch replaces the use of `list.remove` with a `filter` which solves both of the above mentioned problems. PR-URL: #23727 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Trott · 2018-10-22T23:58:38Z

Added a commit to skip the test on Debug builds.

CI: https://ci.nodejs.org/job/node-test-pull-request/18057/

Trott · 2018-10-24T03:59:39Z

CI is green! 🎉

@refack OK if we land this and you or @forivall or someone else can open a PR for the subsequent Makefile changes you suggest above?

refack · 2018-10-24T13:53:50Z

OK if we land this and you or @forivall or someone else can open a PR for the subsequent Makefile changes you suggest above?

Yes 👍
And thank you for following up on my comment.

Also makes sure that the napi benchmark is built before running jstest. Skipped on windows since n-api benchmarks aren't built there yet. PR-URL: nodejs#23585 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Denys Otrishko <shishugi@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

Trott · 2018-10-24T13:59:15Z

Landed in 8c99a22.

Thanks for the contribution! 🎉

(If you're interested in other possible contributions to Node.js but don't have a good idea of where to start looking, some ideas are posted at https://www.nodetodo.org/next-steps/.)

Also makes sure that the napi benchmark is built before running jstest. Skipped on windows since n-api benchmarks aren't built there yet. PR-URL: #23585 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Denys Otrishko <shishugi@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

Ref: #23585 (comment) Python's `list.remove` will throw if the element is not found and also it removes only the first occurrence. This patch replaces the use of `list.remove` with a `filter` which solves both of the above mentioned problems. PR-URL: #23727 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Also makes sure that the napi benchmark is built before running jstest. Skipped on windows since n-api benchmarks aren't built there yet. PR-URL: #23585 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Denys Otrishko <shishugi@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

Ref: #23585 (comment) Python's `list.remove` will throw if the element is not found and also it removes only the first occurrence. This patch replaces the use of `list.remove` with a `filter` which solves both of the above mentioned problems. PR-URL: #23727 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Also makes sure that the napi benchmark is built before running jstest. Skipped on windows since n-api benchmarks aren't built there yet. PR-URL: #23585 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Denys Otrishko <shishugi@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

Ref: #23585 (comment) Python's `list.remove` will throw if the element is not found and also it removes only the first occurrence. This patch replaces the use of `list.remove` with a `filter` which solves both of the above mentioned problems. PR-URL: #23727 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Also makes sure that the napi benchmark is built before running jstest. Skipped on windows since n-api benchmarks aren't built there yet. PR-URL: #23585 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Denys Otrishko <shishugi@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

test: add test-benchmark-napi

b22ee20

also makes sure that the napi benchmark is built before running jstest skipped on windows since n-api benchmarks aren't built there yet

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Oct 12, 2018

targos added the code-and-learn Issues related to the Code-and-Learn events and PRs submitted during the events. label Oct 12, 2018

jasnell approved these changes Oct 12, 2018

View reviewed changes

addaleax approved these changes Oct 12, 2018

View reviewed changes

lundibundi approved these changes Oct 12, 2018

View reviewed changes

addaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 12, 2018

BridgeAR approved these changes Oct 12, 2018

View reviewed changes

BridgeAR reviewed Oct 12, 2018

View reviewed changes

forivall and others added 2 commits October 12, 2018 17:47

fixup: for ci

c62111c

Update test-benchmark-napi.js

e7ca5ef

gireeshpunathil approved these changes Oct 13, 2018

View reviewed changes

thefourtheye approved these changes Oct 13, 2018

View reviewed changes

fixup: use singlequote for strings (linter)

8309f44

BridgeAR removed the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 15, 2018

richardlau reviewed Oct 17, 2018

View reviewed changes

Trott force-pushed the forivall/test-benchmark-napi branch from 0e67357 to 8309f44 Compare October 17, 2018 04:46

fixup! fixup: use singlequote for strings (linter)

81bf8ba

fixup! fixup! fixup! fixup: use singlequote for strings (linter)

8ef164e

refack added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 23, 2018

Trott closed this Oct 24, 2018

targos mentioned this pull request Oct 27, 2018

Release proposal: v11.1.0 #23922

Merged

MylesBorins added dont-land-on-v8.x labels Nov 26, 2018

codebytere mentioned this pull request Nov 27, 2018

v10.13.1 proposal #24675

Closed

codebytere mentioned this pull request Nov 29, 2018

v10.14.2 proposal #24727

Merged

Uh oh!

Conversation

forivall commented Oct 12, 2018

Checklist

Uh oh!

addaleax commented Oct 12, 2018

Uh oh!

BridgeAR Oct 12, 2018

Choose a reason for hiding this comment

Uh oh!

Trott commented Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Trott commented Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gireeshpunathil commented Oct 13, 2018

Uh oh!

forivall commented Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Trott commented Oct 13, 2018

Uh oh!

Trott commented Oct 14, 2018

Uh oh!

richardlau commented Oct 14, 2018

Uh oh!

richardlau commented Oct 14, 2018

Uh oh!

NickNaso commented Oct 14, 2018

Uh oh!

Trott commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

richardlau Oct 17, 2018

Choose a reason for hiding this comment

Uh oh!

Trott Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richardlau Oct 17, 2018

Choose a reason for hiding this comment

Uh oh!

Trott Oct 17, 2018

Choose a reason for hiding this comment

Uh oh!

Trott commented Oct 17, 2018

Uh oh!

Trott commented Oct 19, 2018

Uh oh!

Trott commented Oct 20, 2018

Uh oh!

richardlau commented Oct 20, 2018

Uh oh!

refack commented Oct 20, 2018

Uh oh!

Trott commented Oct 22, 2018

Uh oh!

Trott commented Oct 24, 2018

Uh oh!

refack commented Oct 24, 2018

Uh oh!

Trott commented Oct 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

Trott commented Oct 13, 2018 •

edited

Loading

Trott commented Oct 13, 2018 •

edited

Loading

forivall commented Oct 13, 2018 •

edited

Loading

Trott commented Oct 17, 2018 •

edited

Loading

Trott Oct 17, 2018 •

edited

Loading