http: overridable keep-alive behavior of `Agent` by indutny · Pull Request #13005 · nodejs/node

indutny · 2017-05-12T21:00:49Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

http

Introduce two overridable Agent methods:

keepSocketAlive(socket)
reuseSocket(socket, req)

These methods can be overridden by particular Agent class child to
make keep-alive behavior customizable.

Motivation: destroy persisted sockets after some configurable timeout.
It is very non-trivial to do it with available primitives. Such program
will most likely need to poke with undocumented events and methods of
Agent. With introduced API such behavior is easy to implement.

indutny · 2017-05-12T21:01:21Z

cc @nodejs/http @nodejs/collaborators

@nodejs/ctc this is mostly a proposal for a new API, so your attention is very desired.

refack · 2017-05-12T21:06:33Z

Is this much of a performance boost. I thought creating sockets was cheap(ish)...

indutny · 2017-05-12T21:16:21Z

@refack this doesn't change performance in any way. Just a way to override defaults.

refack · 2017-05-12T21:35:04Z

I assume that reusing the socket has some benefit?

indutny · 2017-05-12T21:37:49Z

@refack it has many benefits: congestion improvement, DNS lookup time economy, SYN+ACK time saving.

refack · 2017-05-12T21:47:08Z

Got it. You're just exposing internal keep-alive semantics for overriding...

mscdex · 2017-05-13T00:10:59Z

test/parallel/test-http-keepalive-override.js

Wrap this in common.mustCall(...., 3) and remove the two unnecessary assertions (one in here and at the end)?

mscdex · 2017-05-13T00:11:27Z

test/parallel/test-http-keepalive-override.js

Ditto, wrap in common.mustCall().

mscdex · 2017-05-13T03:53:59Z

test/parallel/test-http-keepalive-override.js

Both of these variables are unused

Overlooked, thanks!

mscdex · 2017-05-13T04:33:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/8055/

mcollina

LGTM

cjihrig · 2017-05-15T14:07:55Z

lib/_http_agent.js

Why is req passed in here? It appears to be unused. Is it for agents that override the function and want to use it?

Yes, this is correct.

jasnell

Couple of doc nits. Otherwise looks fine to me.

jasnell · 2017-05-15T14:39:42Z

doc/api/http.md

returns a falsy value

Nit: Dash (-) should be changed to a comma (,).

jasnell · 2017-05-15T14:40:24Z

doc/api/http.md

this should list the arguments... e.g.

* `socket` {net.Socket|tls.TLSSocket} description...

jasnell · 2017-05-15T14:41:05Z

doc/api/http.md

as above... this should list the arguments...

* `socket` {net.Socket|tls.TLSSocket} ... * `req` ...

rvagg · 2017-05-24T11:03:57Z

+1

rvagg · 2017-05-24T12:25:16Z

Brought up in CTC meeting today, nothing was added there, no objections and I voiced my support for this. Unless there's an objection brought up here I don't see why this couldn't move forward.

jasnell · 2017-05-24T14:22:58Z

I do have one outstanding comment that has not been addressed: #13005 (comment)

mikeal · 2017-05-25T17:19:11Z

Can we get a citgm run on this @MylesBorins? This is a historically brittle area of code where modules tend to depend on esoteric and undocumented internal semantics.

refack · 2017-05-25T17:35:31Z

CITGM CI: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/830/

refack

CITGM is clean.
Rubber stamp code.

mscdex · 2017-05-25T22:01:23Z

doc/api/http.md

s/by/by a/

Also, I'm not sure that "Agent class child" is the best terminology? What about 'subclass' or 'implementation'?

Introduce two overridable `Agent` methods: * `keepSocketAlive(socket)` * `reuseSocket(socket, req)` These methods can be overridden by particular `Agent` class child to make keep-alive behavior customizable. Motivation: destroy persisted sockets after some configurable timeout. It is very non-trivial to do it with available primitives. Such program will most likely need to poke with undocumented events and methods of `Agent`. With introduced API such behavior is easy to implement. PR-URL: #13005 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>

indutny · 2017-08-02T01:55:55Z

@nodejs/lts was it ever backported to v6.x?

MylesBorins · 2017-08-02T15:01:55Z

@indutny not yet, it will land in a future semver minor

MylesBorins · 2017-09-19T21:21:47Z

LTS WG agreed that this can be packported

MylesBorins · 2018-01-11T18:08:26Z

@indutny this will need to be manually backported. Would you be able to open one?

Introduce two overridable `Agent` methods: * `keepSocketAlive(socket)` * `reuseSocket(socket, req)` These methods can be overridden by particular `Agent` class child to make keep-alive behavior customizable. Motivation: destroy persisted sockets after some configurable timeout. It is very non-trivial to do it with available primitives. Such program will most likely need to poke with undocumented events and methods of `Agent`. With introduced API such behavior is easy to implement. Backport-PR-URL: #18168 PR-URL: #13005 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>

This LTS release comes with 109 commits, 17 of which are considered Semver-Minor. This includes 32 which are doc related, 29 which are test related, 8 which are build / tool related and 1 commit which updates a dependency. Notable Changes: * console: - added console.count() and console.clear() (James M Snell) #12678 * crypto: - expose ECDH class (Bryan English) #8188 - added cypto.randomFill() and crypto.randomFillSync() (Evan Lucas) #10209 - warn on invalid authentication tag length (Tobias Nießen) #17566 * deps: - upgrade libuv to 1.16.1 (cjihrig) #16835 * dgram: - added socket.setMulticastInterface() (Will Young) #7855 * http: - add agent.keepSocketAlive and agent.reuseSocket as to allow overridable keep-alive behavior of `Agent` (Fedor Indutny) #13005 * lib: - return this from net.Socket.end() (Sam Roberts) #13481 * module: - add builtinModules api that provides list of all builtin modules in Node (Jon Moss) #16386 * net: - return this from getConnections() (Sam Roberts) #13553 * promises: - more robust stringification for unhandled rejections (Timothy Gu) #13784 * repl: - improve require() autocompletion (Alexey Orlenko) #14409 * src: - add openssl-system-ca-path configure option (Daniel Bevenius) #16790 - add --use-bundled-ca --use-openssl-ca check (Daniel Bevenius) #12087 - add process.ppid (cjihrig) #16839 * tls: - accept `lookup` option for `tls.connect()` (Fedor Indutny) #12839 * tools, build: - a new macOS installer! (JP Wesselink) #15179 * url: - WHATWG URL api support (James M Snell) #7448 * util: - add %i and %f formatting specifiers (Roman Reiss) #10308 PR-URL: #18342

Introduce two overridable `Agent` methods: * `keepSocketAlive(socket)` * `reuseSocket(socket, req)` These methods can be overridden by particular `Agent` class child to make keep-alive behavior customizable. Motivation: destroy persisted sockets after some configurable timeout. It is very non-trivial to do it with available primitives. Such program will most likely need to poke with undocumented events and methods of `Agent`. With introduced API such behavior is easy to implement. Backport-PR-URL: #18168 PR-URL: #13005 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>

This LTS release comes with 109 commits, 17 of which are considered Semver-Minor. This includes 32 which are doc related, 29 which are test related, 8 which are build / tool related and 1 commit which updates a dependency. Notable Changes: * console: - added console.count() and console.clear() (James M Snell) #12678 * crypto: - expose ECDH class (Bryan English) #8188 - added cypto.randomFill() and crypto.randomFillSync() (Evan Lucas) #10209 - warn on invalid authentication tag length (Tobias Nießen) #17566 * deps: - upgrade libuv to 1.16.1 (cjihrig) #16835 * dgram: - added socket.setMulticastInterface() (Will Young) #7855 * http: - add agent.keepSocketAlive and agent.reuseSocket as to allow overridable keep-alive behavior of `Agent` (Fedor Indutny) #13005 * lib: - return this from net.Socket.end() (Sam Roberts) #13481 * module: - add builtinModules api that provides list of all builtin modules in Node (Jon Moss) #16386 * net: - return this from getConnections() (Sam Roberts) #13553 * promises: - more robust stringification for unhandled rejections (Timothy Gu) #13784 * repl: - improve require() autocompletion (Alexey Orlenko) #14409 * src: - add openssl-system-ca-path configure option (Daniel Bevenius) #16790 - add --use-bundled-ca --use-openssl-ca check (Daniel Bevenius) #12087 - add process.ppid (cjihrig) #16839 * tls: - accept `lookup` option for `tls.connect()` (Fedor Indutny) #12839 * tools, build: - a new macOS installer! (JP Wesselink) #15179 * url: - WHATWG URL api support (James M Snell) #7448 * util: - add %i and %f formatting specifiers (Roman Reiss) #10308 PR-URL: #18342

Introduce two overridable `Agent` methods: * `keepSocketAlive(socket)` * `reuseSocket(socket, req)` These methods can be overridden by particular `Agent` class child to make keep-alive behavior customizable. Motivation: destroy persisted sockets after some configurable timeout. It is very non-trivial to do it with available primitives. Such program will most likely need to poke with undocumented events and methods of `Agent`. With introduced API such behavior is easy to implement. Backport-PR-URL: #18168 PR-URL: #13005 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>

This LTS release comes with 109 commits, 17 of which are considered Semver-Minor. This includes 32 which are doc related, 29 which are test related, 8 which are build / tool related and 1 commit which updates a dependency. Notable Changes: * console: - added console.count() and console.clear() (James M Snell) #12678 * crypto: - expose ECDH class (Bryan English) #8188 - added cypto.randomFill() and crypto.randomFillSync() (Evan Lucas) #10209 - warn on invalid authentication tag length (Tobias Nießen) #17566 * deps: - upgrade libuv to 1.16.1 (cjihrig) #16835 * dgram: - added socket.setMulticastInterface() (Will Young) #7855 * http: - add agent.keepSocketAlive and agent.reuseSocket as to allow overridable keep-alive behavior of `Agent` (Fedor Indutny) #13005 * lib: - return this from net.Socket.end() (Sam Roberts) #13481 * module: - add builtinModules api that provides list of all builtin modules in Node (Jon Moss) #16386 * net: - return this from getConnections() (Sam Roberts) #13553 * promises: - more robust stringification for unhandled rejections (Timothy Gu) #13784 * repl: - improve require() autocompletion (Alexey Orlenko) #14409 * src: - add openssl-system-ca-path configure option (Daniel Bevenius) #16790 - add --use-bundled-ca --use-openssl-ca check (Daniel Bevenius) #12087 - add process.ppid (cjihrig) #16839 * tls: - accept `lookup` option for `tls.connect()` (Fedor Indutny) #12839 * tools, build: - a new macOS installer! (JP Wesselink) #15179 * url: - WHATWG URL api support (James M Snell) #7448 * util: - add %i and %f formatting specifiers (Roman Reiss) #10308 PR-URL: #18342

This LTS release comes with 112 commits, 17 of which are considered Semver-Minor. This includes 32 which are doc related, 30 which are test related, 8 which are build / tool related and 1 commit which updates a dependency. Notable Changes: * console: - added console.count() and console.clear() (James M Snell) #12678 * crypto: - expose ECDH class (Bryan English) #8188 - added cypto.randomFill() and crypto.randomFillSync() (Evan Lucas) #10209 - warn on invalid authentication tag length (Tobias Nießen) #17566 * deps: - upgrade libuv to 1.16.1 (cjihrig) #16835 * dgram: - added socket.setMulticastInterface() (Will Young) #7855 * http: - add agent.keepSocketAlive and agent.reuseSocket as to allow overridable keep-alive behavior of `Agent` (Fedor Indutny) #13005 * lib: - return this from net.Socket.end() (Sam Roberts) #13481 * module: - add builtinModules api that provides list of all builtin modules in Node (Jon Moss) #16386 * net: - return this from getConnections() (Sam Roberts) #13553 * promises: - more robust stringification for unhandled rejections (Timothy Gu) #13784 * repl: - improve require() autocompletion (Alexey Orlenko) #14409 * src: - add openssl-system-ca-path configure option (Daniel Bevenius) #16790 - add --use-bundled-ca --use-openssl-ca check (Daniel Bevenius) #12087 - add process.ppid (cjihrig) #16839 * tls: - accept `lookup` option for `tls.connect()` (Fedor Indutny) #12839 * tools, build: - a new macOS installer! (JP Wesselink) #15179 * url: - WHATWG URL api support (James M Snell) #7448 * util: - add %i and %f formatting specifiers (Roman Reiss) #10308 PR-URL: #18342

This LTS release comes with 112 commits, 17 of which are considered Semver-Minor. This includes 32 which are doc related, 30 which are test related, 8 which are build / tool related and 1 commit which updates a dependency. Notable Changes: * console: - added console.count() and console.clear() (James M Snell) nodejs#12678 * crypto: - expose ECDH class (Bryan English) nodejs#8188 - added cypto.randomFill() and crypto.randomFillSync() (Evan Lucas) nodejs#10209 - warn on invalid authentication tag length (Tobias Nießen) nodejs#17566 * deps: - upgrade libuv to 1.16.1 (cjihrig) nodejs#16835 * dgram: - added socket.setMulticastInterface() (Will Young) nodejs#7855 * http: - add agent.keepSocketAlive and agent.reuseSocket as to allow overridable keep-alive behavior of `Agent` (Fedor Indutny) nodejs#13005 * lib: - return this from net.Socket.end() (Sam Roberts) nodejs#13481 * module: - add builtinModules api that provides list of all builtin modules in Node (Jon Moss) nodejs#16386 * net: - return this from getConnections() (Sam Roberts) nodejs#13553 * promises: - more robust stringification for unhandled rejections (Timothy Gu) nodejs#13784 * repl: - improve require() autocompletion (Alexey Orlenko) nodejs#14409 * src: - add openssl-system-ca-path configure option (Daniel Bevenius) nodejs#16790 - add --use-bundled-ca --use-openssl-ca check (Daniel Bevenius) nodejs#12087 - add process.ppid (cjihrig) nodejs#16839 * tls: - accept `lookup` option for `tls.connect()` (Fedor Indutny) nodejs#12839 * tools, build: - a new macOS installer! (JP Wesselink) nodejs#15179 * url: - WHATWG URL api support (James M Snell) nodejs#7448 * util: - add %i and %f formatting specifiers (Roman Reiss) nodejs#10308 PR-URL: nodejs#18342

Introduce two overridable `Agent` methods: * `keepSocketAlive(socket)` * `reuseSocket(socket, req)` These methods can be overridden by particular `Agent` class child to make keep-alive behavior customizable. Motivation: destroy persisted sockets after some configurable timeout. It is very non-trivial to do it with available primitives. Such program will most likely need to poke with undocumented events and methods of `Agent`. With introduced API such behavior is easy to implement. PR-URL: nodejs/node#13005 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>

nodejs-github-bot added the http Issues or PRs related to the http subsystem. label May 12, 2017

indutny requested review from jasnell and sam-github May 12, 2017 21:00

mscdex reviewed May 13, 2017

View reviewed changes

test/parallel/test-http-keepalive-override.js Outdated

Copy link

Contributor

mscdex May 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto, wrap in common.mustCall().

mscdex reviewed May 13, 2017

View reviewed changes

mcollina approved these changes May 15, 2017

View reviewed changes

cjihrig reviewed May 15, 2017

View reviewed changes

jasnell requested changes May 15, 2017

View reviewed changes

indutny added the ctc-agenda label May 17, 2017

indutny mentioned this pull request May 17, 2017

Node.js Foundation Core Technical Committee (CTC) Meeting 2017-05-17 nodejs/CTC#127

Closed

Trott mentioned this pull request May 24, 2017

Node.js Foundation Core Technical Committee (CTC) Meeting 2017-05-24 nodejs/CTC#130

Closed

bnoordhuis removed the ctc-agenda label May 24, 2017

targos added the ctc-review label May 24, 2017

refack approved these changes May 24, 2017

View reviewed changes

cjihrig approved these changes May 24, 2017

View reviewed changes

refack approved these changes May 25, 2017

View reviewed changes

mscdex reviewed May 25, 2017

View reviewed changes

gibfahn mentioned this pull request Jun 7, 2017

Review semver-minor backports nodejs/Release#228

Closed

8 tasks

Trott removed the ctc-review label Jun 11, 2017

MylesBorins added the backport-requested-v6.x label Jan 15, 2018

indutny mentioned this pull request Jan 16, 2018

[v6.x] http: overridable keep-alive behavior of Agent #18168

Closed

4 tasks

cjihrig mentioned this pull request Jan 16, 2018

Investigate flaky parallel/test-dgram-multicast-set-interface #18193

Closed

MylesBorins mentioned this pull request Jan 24, 2018

v6.13.0 proposal #18342

Merged

Uh oh!

Conversation

indutny commented May 12, 2017

Checklist

Affected core subsystem(s)

Uh oh!

indutny commented May 12, 2017

Uh oh!

refack commented May 12, 2017

Uh oh!

indutny commented May 12, 2017

Uh oh!

refack commented May 12, 2017

Uh oh!

indutny commented May 12, 2017

Uh oh!

refack commented May 12, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mscdex May 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mscdex commented May 13, 2017

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasnell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Trott May 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rvagg commented May 24, 2017

Uh oh!

rvagg commented May 24, 2017

Uh oh!

jasnell commented May 24, 2017

Uh oh!

mikeal commented May 25, 2017

Uh oh!

refack commented May 25, 2017

Uh oh!

refack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

indutny commented Aug 2, 2017

Uh oh!

MylesBorins commented Aug 2, 2017

Uh oh!

MylesBorins commented Sep 19, 2017

Uh oh!

MylesBorins commented Jan 11, 2018

mscdex May 13, 2017 •

edited

Loading

Trott May 21, 2017 •

edited

Loading