<(target_arch) not resolving, causing npm install to fail

[yeerkkiller1]

• Node Version:
  node: v10.6.0, npm: 6.1.0
• Platform:
  MSYS_NT-10.0 DESKTOP-E7DLHGV 2.8.2(0.313/5/3) 2017-07-12 15:35 x86_64 Msys
• Compiler:
  Microsoft (R) Build Engine version 15.6.85.37198 for .NET Framework
  Copyright (C) Microsoft Corporation. All rights reserved.
  15.6.85.37198
  Microsoft (R) C/C++ Optimizing Compiler Version 19.13.26131.1 for x86
  Copyright (C) Microsoft Corporation. All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

• Module:
  https://github.com/nodejs/abi-stable-node-addon-examples/blob/master/1_hello_world/napi/hello.cc

npm install

C:\Users\quent\Dropbox\compile-test>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
The system cannot find the path specified.
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:336:16)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:237:12)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Users\quent\Dropbox\compile-test
gyp ERR! node -v v10.6.0
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! compile-test@0.0.1 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the compile-test@0.0.1 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\quent\AppData\Roaming\npm-cache\_logs\2018-07-14T05_29_31_347Z-debug.log

node "C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node-gyp\bin\node-gyp.js" rebuild

gyp info it worked if it ends with ok
gyp info using node-gyp@3.6.2
gyp info using node@10.6.0 | win32 | x64
gyp info spawn C:\src\depot_tools\python.BAT
gyp info spawn args [ 'C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'msvs',
The system cannot find the path specified.
gyp info spawn args   '-G',
gyp info spawn args   'msvs_version=2015',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\quent\\Dropbox\\compile-test\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\quent\\.node-gyp\\10.6.0\\include\\node\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=C:\\Users\\quent\\.node-gyp\\10.6.0',
gyp info spawn args   '-Dnode_gyp_dir=C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=C:\\Users\\quent\\.node-gyp\\10.6.0\\<(target_arch)\\node.lib',
gyp info spawn args   '-Dmodule_root_dir=C:\\Users\\quent\\Dropbox\\compile-test',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'C:\\Users\\quent\\Dropbox\\compile-test\\build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:336:16)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:237:12)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Users\quent\Dropbox\compile-test
gyp ERR! node -v v10.6.0
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok

​​

The specific problem is <(target_arch), which comes from this line: https://github.com/nodejs/node-gyp/blob/master/lib/configure.js#L296 .

It appears as if <(target_arch) (process substitution) is being used, which I believe is only valid syntax on linux? I am not very familiar with shell scripting, but it looks like $(target_arch) would work? And at the very least be preferable on windows?

But there could also be another issue causing <(target_arch) to not work on my machine? Because it looks like that line is years old.

[bzoz]

The <(target_arch) is not a problem, here (gist) is a log from successful build.

Could you try running npm install --verbose and post the log that it produces?

[bzoz]

One more thing: in the log I see C:\src\depot_tools\python.BAT. It is not the first time that I see Python installed this way and issues with node-gyp. Could you describe how did you set up your build environment?

[yeerkkiller1]

Hi, thanks for the investigation! I didn't even notice C:\src\depot_tools\python.BAT.

C:\src\depot_tools is installed as part of building chromium, from their instructions here: https://chromium.googlesource.com/chromium/src/+/master/docs/windows_build_instructions.md.

`npm install --verbose`

C:\Users\quent\Dropbox\compile-test>if not defined npm_config_node_gyp (node "C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\npm-lifecycle\node_modules\node-gyp\bin\node-gyp.js" rebuild )
gyp info it worked if it ends with ok
gyp verb cli [ 'C:\\Program Files\\nodejs\\node.exe',
gyp verb cli   'C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\npm-lifecycle\\node_modules\\node-gyp\\bin\\node-gyp.js',
gyp verb cli   'rebuild' ]
gyp info using node-gyp@3.7.0
gyp info using node@10.6.0 | win32 | x64
gyp verb command rebuild []
gyp verb command clean []
gyp verb clean removing "build" directory
gyp verb command configure []
gyp verb check python checking for Python executable "python2" in the PATH
gyp verb `which` failed Error: not found: python2
gyp verb `which` failed     at getNotFoundError (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\which.js:13:12)
gyp verb `which` failed     at F (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\which.js:68:19)
gyp verb `which` failed     at E (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\which.js:80:29)
gyp verb `which` failed     at C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\which.js:89:16
gyp verb `which` failed     at C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\node_modules\isexe\index.js:42:5
gyp verb `which` failed     at C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\node_modules\isexe\windows.js:36:5
gyp verb `which` failed     at FSReqWrap.oncomplete (fs.js:158:21)
gyp verb `which` failed  python2 { Error: not found: python2
gyp verb `which` failed     at getNotFoundError (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\which.js:13:12)
gyp verb `which` failed     at F (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\which.js:68:19)
gyp verb `which` failed     at E (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\which.js:80:29)
gyp verb `which` failed     at C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\which.js:89:16
gyp verb `which` failed     at C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\node_modules\isexe\index.js:42:5
gyp verb `which` failed     at C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\which\node_modules\isexe\windows.js:36:5
gyp verb `which` failed     at FSReqWrap.oncomplete (fs.js:158:21)
gyp verb `which` failed   stack:
gyp verb `which` failed    'Error: not found: python2\n    at getNotFoundError (C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\which\\which.js:13:12)\n    at F (C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\which\\which.js:68:19)\n    at E (C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\which\\which.js:80:29)\n    at C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\which\\which.js:89:16\n    at C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\which\\node_modules\\isexe\\index.js:42:5\n    at C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\which\\node_modules\\isexe\\windows.js:36:5\n    at FSReqWrap.oncomplete (fs.js:158:21)',
gyp verb `which` failed   code: 'ENOENT' }
gyp verb check python checking for Python executable "python" in the PATH
gyp verb `which` succeeded python C:\path\python.BAT
gyp verb check python version `C:\path\python.BAT -c "import platform; print(platform.python_version());"` returned: "2.7.6\r\n"
gyp verb get node dir no --target version specified, falling back to host node version: 10.6.0
gyp verb command install [ '10.6.0' ]
gyp verb install input version string "10.6.0"
gyp verb install installing version: 10.6.0
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version is already installed, need to check "installVersion"
gyp verb got "installVersion" 9
gyp verb needs "installVersion" 9
gyp verb install version is good
gyp verb get node dir target node version installed: 10.6.0
gyp verb build dir attempting to create "build" dir: C:\Users\quent\Dropbox\compile-test\build
gyp verb build dir "build" dir needed to be created? C:\Users\quent\Dropbox\compile-test\build
gyp verb find vs2017 Found installation at: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community
gyp verb find vs2017   - Found Microsoft.VisualStudio.Component.Windows10SDK.16299.Desktop
gyp verb find vs2017   - Found Microsoft.VisualStudio.Component.VC.Tools.x86.x64
gyp verb find vs2017   - Found Microsoft.VisualStudio.VC.MSBuild.Base
gyp verb find vs2017   - Found Microsoft.VisualStudio.Component.Windows10SDK.15063.Desktop
gyp verb find vs2017   - Using this installation with Windows 10 SDK
gyp verb find vs2017 using installation: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community
gyp verb build/config.gypi creating config file
gyp verb build/config.gypi writing out config file: C:\Users\quent\Dropbox\compile-test\build\config.gypi
gyp verb config.gypi checking for gypi file: C:\Users\quent\Dropbox\compile-test\config.gypi
gyp verb common.gypi checking for gypi file: C:\Users\quent\Dropbox\compile-test\common.gypi
gyp verb gyp gyp format was not specified; forcing "msvs"
gyp info spawn C:\path\python.BAT
gyp info spawn args [ 'C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\npm-lifecycle\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
�[0mgyp info spawn args   '-f',
gyp info spawn args   'msvs',
gyp info The system cannot find the path specified.
spawn args   '-G',
gyp info spawn args   'msvs_version=2015',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\quent\\Dropbox\\compile-test\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\npm-lifecycle\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\quent\\.node-gyp\\10.6.0\\include\\node\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=C:\\Users\\quent\\.node-gyp\\10.6.0',
gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\npm-lifecycle\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=C:\\Users\\quent\\.node-gyp\\10.6.0\\<(target_arch)\\node.lib',
gyp info spawn args   '-Dmodule_root_dir=C:\\Users\\quent\\Dropbox\\compile-test',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'C:\\Users\\quent\\Dropbox\\compile-test\\build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\npm-lifecycle\node_modules\node-gyp\lib\configure.js:346:16)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:237:12)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\quent\\AppData\\Roaming\\nvm\\v10.6.0\\node_modules\\npm\\node_modules\\npm-lifecycle\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Users\quent\Dropbox\compile-test
gyp ERR! node -v v10.6.0
gyp ERR! node-gyp -v v3.7.0
gyp ERR! not ok
npm verb lifecycle compile-test@0.0.1~install: unsafe-perm in lifecycle true
npm verb lifecycle compile-test@0.0.1~install: PATH: C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;C:\Users\quent\Dropbox\compile-test\node_modules\.bin;C:\Program Files\ImageMagick-7.0.8-Q16;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\dotnet\;C:\path;C:\Program Files (x86)\GitExtensions\;C:\Go\bin;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Yarn\bin\;C:\Program Files\Microsoft VS Code\bin;C:\Users\quent\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\Program Files\CMake\bin;%GSTREAMER_ROOT_X86%\bin;C:\MinGW\msys\1.0\bin;C:\MinGW\bin;C:\Python2.7.6.1;C:\Users\quent\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Microsoft VS Code\bin;C:\Program Files\GPAC;C:\Users\quent\AppData\Local\Yarn\bin;C:\Users\quent\AppData\Roaming\npm;C:\Users\quent\AppData\Roaming\nvm;C:\Program Files\nodejs;
npm verb lifecycle compile-test@0.0.1~install: CWD: C:\Users\quent\Dropbox\compile-test
npm info lifecycle compile-test@0.0.1~install: Failed to exec install script
npm verb stack Error: compile-test@0.0.1 install: `node-gyp rebuild`
npm verb stack Exit status 1
npm verb stack     at EventEmitter.<anonymous> (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\npm-lifecycle\index.js:304:16)
npm verb stack     at EventEmitter.emit (events.js:182:13)
npm verb stack     at ChildProcess.<anonymous> (C:\Users\quent\AppData\Roaming\nvm\v10.6.0\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
npm verb stack     at ChildProcess.emit (events.js:182:13)
npm verb stack     at maybeClose (internal/child_process.js:961:16)
npm verb stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:5)
npm verb pkgid compile-test@0.0.1
npm verb cwd C:\Users\quent\Dropbox\compile-test
npm verb Windows_NT 10.0.17134
npm verb argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "--verbose"
npm verb node v10.6.0
npm verb npm  v6.1.0
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! compile-test@0.0.1 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the compile-test@0.0.1 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm verb exit [ 1, true ]
npm timing audit submit Completed in 1005ms
npm http fetch POST 200 https://registry.npmjs.org/-/npm/v1/security/audits/quick 880ms
npm timing audit body Completed in 0ms
npm timing npm Completed in 1361ms
npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\quent\AppData\Roaming\npm-cache\_logs\2018-07-19T00_00_53_572Z-debug.log

​​

When I was testing before I made a fork of node-gyp with the <(target_arch) line changed and it fixed my building issues.

However, removing C:\src\depot_tools from my path and adding a regular python 2.7.15 executable to my path also fixed the build issues.

I found I could reproduce the issue simply by redirecting python from a batch file (putting python.bat in my path, instead of python.exe). For example, with (and the @echo off is important, or else the python version detection code in configure.js breaks) this as my batch file:

@echo off "C:\Python2.7.6.1\python.exe" %*

So it looks like using a batch file to redirect inputs breaks when there is a "<" in the arguments.

Works:
node -p "require('child_process').execFileSync('C:/Python2.7.6.1/python', ['-c', 'import sys; print sys.argv', '<(target_arch)']).toString()"

Fails:
node -p "require('child_process').execFileSync('C:/path/python.BAT', ['-c', 'import sys; print sys.argv', '<(target_arch)']).toString()"

Fails:
node -p "require('child_process').execFileSync('C:/path/python.BAT', ['-c', 'import sys; print sys.argv', 'test<']).toString()"

Works:
node -p "require('child_process').execFileSync('C:/path/python.BAT', ['-c', 'import sys; print sys.argv', 'test']).toString()"

Although it sort of seems like this is a chromium problem, if there is only one line we using "<" in an argument it would completely get rid of this type of problem in the future by changing that one line.

It is also possible to check for python not accepting "<" in arguments, which would require developers to change their PATH when switching between chromium source development, and development that requires node-gyp.

I'm going to keep investigating the "<" issue, and see if there is a way chromium can change their python.BAT file to work with "<" arguments.

[refack]

Hi @yeerkkiller1. Thanks for the informative submission.
AFAIK the <(target_arch) construct is used here as a GYP var definition, not as a shell command. And that just might be the problem. C:/Python2.7.6.1/python.exe is an executable file, and execFileSync invokes it directly. On the other hand C:/path/python.BAT is a CMD batch file so execFileSync has to invoke it via a shell... (not sure, but I have a strong suspicion...)

Simplest solution might be to write node_lib_file=C:\\Users\\quent\\.node-gyp\\10.6.0\\<(target_arch)\\node.lib to config.gypi

[yeerkkiller1]

Hi @refack. Yep, it looks like that is the problem. I looked up escaping in windows BATCH files and found that '^^^<(target_arch)' fixes it. I made a pull request to use this string (when on windows, and python is a batch file), here: #1506 .

Thanks @refack and @bzoz !