Reduce likelihood of race conditions on keep-alive timeout calculatio… by zanettea · Pull Request #52653 · nodejs/node

zanettea · 2024-04-23T10:48:33Z

Added 1 seconds threshold in keepalive timeout client-side
Added 1 second threshold in keepalive timeout server-side (expire the socket timeout 1 sec after the announced timeout)

Probably better to use a configurable threshold like in undici keepAliveTimeoutThreshold (nodejs/undici#291)

nodejs-github-bot · 2024-04-23T10:48:39Z

Review requested:

@nodejs/http
@nodejs/net

mcollina

lgtm

mcollina · 2024-04-23T10:52:54Z

ping @mweberxyz

nodejs-github-bot · 2024-04-23T10:54:29Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58633/

mweberxyz · 2024-04-23T14:30:51Z

lib/_http_agent.js

+        let serverHintTimeout = ( NumberParseInt(hint) * 1000 ) - 1000;
+        serverHintTimeout = serverHintTimeout > 0 ? serverHintTimeout : 0;

        if (serverHintTimeout < agentTimeout) {


This conditional needs to be fixed - agentTimeout defaults to 0, so the serverHintTimeout is never being set.

In this case the socket shouldn't be reused at all. Is setting socket.setTimeout(0) the right way to do it?

Take a look at socket docs - socket.setTimeout(0) doesn't mean immediate, it means never.

Sorry, I have limited understanding of the network internals. In undici ( https://github.com/nodejs/undici/pull/291/files ) if the keepAliveTimeout goes down to 0 they flag the connection as reset:

if (!keepAliveTimeout || keepAliveTimeout < 1e3) { client[kReset] = true }

I don't know how to achieve the same result here. Maybe:

socket.setKeepAlive(false);

?

I also have a limited understanding of network internals, but I think:

if (!agentTimeout || serverHintTimeout < agentTimeout) {

is what you want.

mweberxyz · 2024-04-23T14:36:45Z

lib/_http_server.js

-      socket.setTimeout(server.keepAliveTimeout);
+      // Increase the internal timeout wrt the advertised value to reduce likeliwood of ECONNRESET errors
+      // due to race conditions between the client and server timeout calculation
+      socket.setTimeout(server.keepAliveTimeout + 1000);


I don't think this server change is necessary.

Might be the cause of all the test failures.

This fix protects clients that are using the hint timeout as is (current node impl) without adjusting it for network (or cpu load) delays

If server.keepAliveTimeout is set to 0 (never time out) this will change it to 1 second.

If server.keepAliveTimeout == 0 then the if condition line1012 is false. The timeout on the socket is set only if server.keepAliveTimeout != 0

You're correct, sorry I missed that.

mweberxyz · 2024-04-23T16:03:01Z

lib/_http_agent.js

-        const serverHintTimeout = NumberParseInt(hint) * 1000;
+        // Let the timer expires before the announced timeout to reduce
+        // the likelihood of ECONNRESET errors
+        let serverHintTimeout = ( NumberParseInt(hint) * 1000 ) - 1000;


In the case the server responds with a 1 second keepalive, this will set the disable the timeout because serverHintTimeout will be 0 -- maybe go with - 500 in place of - 1000 ?

zanettea · 2024-04-23T18:10:20Z

lib/_http_agent.js

+        serverHintTimeout = serverHintTimeout > 0 ? serverHintTimeout : 0;
+        if (serverHintTimeout === 0) {
+          // cannot safely reuse the socket because the server timeout is too short
+          canKeepSocketAlive = false;


I have updated the PR to just skip keep alive if the timeout is too short. In my local build the test now pass.

lib/_http_agent.js

mweberxyz · 2024-04-24T13:59:34Z

Found the same issue in dotnet -- they went with a 1 second offset as well, so that was a good choice. 👍

mweberxyz

Thanks for the updates! I verified that the test cases in #52649 pass without error.

I don't have Approve permission - lgtm @mcollina

ShogunPanda

LGTM!

reduce likelihood of race conditions on keep-alive timeout calculation between http1.1 servers and clients and honor server keep-alive timeout when agentTimeout is not set Fixes: nodejs#47130 Fixes: nodejs#52649

fahrradflucht

To me, it looks like this should be a testable behavior change.

I had a short look and I guess if test-http-client-keep-alive-hint.js would be a better test it should actually fail now, because the case it tests now behaves quite different (there is no keep-alive because the hint is just 1 second).

I think a test somewhat along these lines would pass on main and fail on this branch:

'use strict';

const common = require('../common');
const assert = require('assert');
const http = require('http');

const server = http.createServer(
  { keepAliveTimeout: common.platformTimeout(60000) },
  function(req, res) {
    req.resume();
    res.writeHead(200, { 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=1' });
    res.end('FOO');
  }
);

server.listen(0, common.mustCall(() => {
  let shouldStillBeAlive = true;
  setTimeout(() => {
    shouldStillBeAlive = false;
  }, common.platformTimeout(500)); // 500ms buffer

  const req = http.get({ port: server.address().port }, (res) => {
    assert.strictEqual(res.statusCode, 200);

    res.resume();
  });

  req.on('socket', (socket) => {
    socket.on('close', common.mustCall(() => {
      if (shouldStillBeAlive) {
        assert.fail('socket prematurely closed');
      }
      server.close();
    }));
  });
}));

// This timer should never go off as the agent will parse the hint and terminate earlier
setTimeout(common.mustNotCall(), common.platformTimeout(3000)).unref();

I guess it would be good to write a similar test (hopefully a version that has to rely less on timers) that verifies that we are actually latency adjusting (currently hard-coded to 1s) the hint based keep alive?

fahrradflucht · 2024-05-19T12:37:24Z

lib/_http_server.js

  } else if (state.outgoing.length === 0) {
    if (server.keepAliveTimeout && typeof socket.setTimeout === 'function') {
-      socket.setTimeout(server.keepAliveTimeout);
+      // Increase the internal timeout wrt the advertised value to reduce likeliwood of ECONNRESET errors


Suggested change

// Increase the internal timeout wrt the advertised value to reduce likeliwood of ECONNRESET errors

// Increase the internal timeout wrt the advertised value to reduce likelihood of ECONNRESET errors

fahrradflucht · 2024-05-19T12:38:12Z

lib/_http_agent.js

-        const serverHintTimeout = NumberParseInt(hint) * 1000;
-
-        if (serverHintTimeout < agentTimeout) {
+        // Let the timer expires before the announced timeout to reduce


Suggested change

// Let the timer expires before the announced timeout to reduce

// Let the timer expire before the announced timeout to reduce

jazelly · 2024-08-28T11:42:35Z

Hey, this valuable PR fixes many similar issues for keep-alive. Is it possible to push this through 👀

nodejs-github-bot · 2024-08-28T11:50:24Z

CI: https://ci.nodejs.org/job/node-test-pull-request/61582/

jakecastelli

Could you please add a test?

jazelly · 2024-09-07T12:39:13Z

Hi, I'd like to help this land to resovle similar issues. Can I continue the work and add @zanettea as a co-author? To me, there are just test cases to be added ~~and some commits to be rebased to dodge the flakiness on CI~~.

nikwen · 2024-09-13T16:17:37Z

The new PR was just merged. This one can be closed now. (Thanks to @jazelly, @zanettea and everyone else who worked on it!)

nodejs-github-bot added http Issues or PRs related to the http subsystem. needs-ci PRs that need a full CI run. labels Apr 23, 2024

mcollina approved these changes Apr 23, 2024

View reviewed changes

mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 23, 2024

mcollina mentioned this pull request Apr 23, 2024

http: keepalive race condition near timeout #52649

Closed

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 23, 2024

mweberxyz reviewed Apr 23, 2024

View reviewed changes

zanettea commented Apr 23, 2024

View reviewed changes

mweberxyz reviewed Apr 24, 2024

View reviewed changes

lib/_http_agent.js Outdated Show resolved Hide resolved

meixg approved these changes Apr 24, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

zanettea changed the title ~~Reduce likeliwood of race conditions on keep-alive timeout calculatio…~~ Reduce likelihood of race conditions on keep-alive timeout calculatio… Apr 24, 2024

zanettea force-pushed the main branch 2 times, most recently from 8407fd7 to bc34680 Compare April 24, 2024 10:12

zanettea force-pushed the main branch 3 times, most recently from ba0d895 to 2124b55 Compare April 24, 2024 15:30

mweberxyz reviewed Apr 24, 2024

View reviewed changes

ShogunPanda approved these changes Apr 26, 2024

View reviewed changes

net: reduce likelihood of race conditions on keep-alive timeout

2c3d383

reduce likelihood of race conditions on keep-alive timeout calculation between http1.1 servers and clients and honor server keep-alive timeout when agentTimeout is not set Fixes: nodejs#47130 Fixes: nodejs#52649

zanettea force-pushed the main branch from 2124b55 to 2c3d383 Compare April 30, 2024 08:04

fahrradflucht reviewed May 19, 2024

View reviewed changes

jazelly mentioned this pull request Aug 28, 2024

Network request fails after event loop is blocked for a few seconds (regression in v20) #54293

Closed

mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 28, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 28, 2024

jakecastelli reviewed Aug 28, 2024

View reviewed changes

jasnell approved these changes Aug 31, 2024

View reviewed changes

jazelly mentioned this pull request Sep 9, 2024

http: reduce likelihood of race conditions on keep-alive timeout #54863

Closed

mcollina closed this Sep 13, 2024

	// Increase the internal timeout wrt the advertised value to reduce likeliwood of ECONNRESET errors
	// Increase the internal timeout wrt the advertised value to reduce likelihood of ECONNRESET errors

	// Let the timer expires before the announced timeout to reduce
	// Let the timer expire before the announced timeout to reduce

Uh oh!

Conversation

zanettea commented Apr 23, 2024

Uh oh!

nodejs-github-bot commented Apr 23, 2024

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

mcollina commented Apr 23, 2024

Uh oh!

nodejs-github-bot commented Apr 23, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zanettea Apr 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mweberxyz Apr 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

mweberxyz commented Apr 24, 2024

Uh oh!

mweberxyz left a comment

Choose a reason for hiding this comment

Uh oh!

ShogunPanda left a comment

Choose a reason for hiding this comment

Uh oh!

fahrradflucht left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jazelly commented Aug 28, 2024

Uh oh!

nodejs-github-bot commented Aug 28, 2024

Uh oh!

jakecastelli left a comment

Choose a reason for hiding this comment

Uh oh!

jazelly commented Sep 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nikwen commented Sep 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

zanettea Apr 23, 2024 •

edited

Loading

mweberxyz Apr 23, 2024 •

edited

Loading

fahrradflucht left a comment •

edited

Loading

jazelly commented Sep 7, 2024 •

edited

Loading