Improve the performance when using enumeration by aplopez · Pull Request #8395 · SSSD/sssd

aplopez · 2026-01-21T14:36:29Z

This PR includes:

Removal of an unused function.
Stop logging a possibly extremely long filter.
Fixes a wrong condition invalidating an optimization.
Adds a test case for an existing test.

Enumeration, specially when there are 15,000+ users, is slow. This fix helps, but it doesn't work miracles.
In my test environment, the enumeration went from 8 minutes to about 1.

It is important to know that, with such an amount of users, many operations time out. It is necessary to increment the timeout in[nss] and for the domain, but also set large values for ldap_enumeration_refresh_timeout and ldap_search_timeout in the domain. I used these values to avoid any timeout (YMMV):

[domain/ldap.test]
ldap_enumeration_refresh_timeout = 30000
ldap_search_timeout = 6000
timeout = 6000
...

[nss]
timeout = 6000
...

gemini-code-assist

Code Review

This pull request effectively improves performance by optimizing logging, removing an unused function, and correcting a condition related to enumeration. The changes are well-aligned with the stated goals of enhancing enumeration performance, especially for large user bases. The addition of a new test case for the general enumeration scenario ensures that the modified logic is adequately covered.

src/db/sysdb_search.c

alexey-tikhonov · 2026-01-22T12:11:25Z

Mistype in the commit message: "We must look into de TS cache"

aplopez · 2026-01-22T15:48:24Z

Mistype in the commit message: "We must look into de TS cache"

Fixed.

alexey-tikhonov · 2026-01-23T09:57:49Z

I think fix is correct in the sense it fixes a bug.

But I think logic of sysdb_enumpwent_filter() can and should be improved in general to avoid a case when dn_filter expands to entire db.

In particular, if addtl_filter isn't set, then sysdb_search_ts_users(enum_filter(NULL)) is expected to return entire db, right? And using this as additional filter results in the same as '*' but extremely slow.
Or do I miss something?

alexey-tikhonov · 2026-02-12T14:57:08Z

src/db/sysdb_search.c

+                                        dn_filter, &ts_cache_res);
+            if (ret != EOK && ret != ENOENT) {
+                goto done;
+            }


Can this go out immediately from else branch?

Which else branch?

if (ts_res.count > 0) {} else {go-out}

I understand we cannot go to out because we still need to proceed with the code that follows the big if {} block.

Why?

search is by name

timestamp cache has nothing for this name pattern

Imo, this means main cache also has nothing. No?
Could you provide an example where ts-search finds nothing but main cache finds something?

It is not simply ignored. It is ignored if no entry is found, but if entries are found, it is used (and only those entries are returned).

In addition, one of the test falls in this case and fails if I add the goto-out as you proposed.
Maybe there is an error in the test setup. I need to confirm this.

It is not simply ignored. It is ignored if no entry is found, but if entries are found, it is used (and only those entries are returned).

Do I understand correctly it returns outdated entries in this case, despite filter clearly set?

Btw, other than this issue discussed in this thread, patch set looks good to me.

It is not simply ignored. It is ignored if no entry is found, but if entries are found, it is used (and only those entries are returned).

Do I understand correctly it returns outdated entries in this case, despite filter clearly set?

Yes. That's what happens. But this happens in known cases. When you call FindByAttr() it will return only new entries. New calls will return newer entries only until there is none, in which case, all the entries will be returned.

Maybe @sumit-bose can explain why?

Function sysdb_enumpwent() is not used. It was replaced by sysdb_enumpwent_filter().

When there are too many users (17,000+) this message can be too long. Limit it to the first 50 characters. Resolves: SSSD#6951

We must look into the TS cache only when a name is provided. Using the TS cache on an unfiltered enumeration is useless. Resolves: SSSD#6951

Added a case that was not checked before. It is the case when `attr`, `attr_name` and `addtl_filter` are all `NULL`.

Create the filter to retrieve only the requested entries. Do not create a new filter and search for matches if there is no results from the previous search. The called functions handle this case correctly but why waisting time calling them?

Function cache_req_user_by_filter_lookup() will set or not the recent filter depending on whether data->name.attr is set or not. As mentioned in the comment, it should be done base on whether the refernced attribute is name or not.

alexey-tikhonov · 2026-02-24T18:15:47Z

Note: Covsan is green so far.

alexey-tikhonov · 2026-02-25T09:01:40Z

Hm,
F44:

FAILED tests/test_infopipe.py::test_infopipe__list_by_name (ldap) - AssertionError: ListByName('user-*', 0) is missing element 10002
assert '/org/freedesktop/sssd/infopipe/Users/test/10002' in ['/org/freedesktop/sssd/infopipe/Users/test/10001', '/org/freedesktop/sssd/infopipe/Users/test/10003']

Looks relevant, but why f44 only... race condition?

aplopez · 2026-02-25T18:17:41Z

Looks relevant, but why f44 only... race condition?

I reran the tests and a different test failed. 😮‍💨
Locally, on my PC (Fedora 43, though) the test passes every time.

aplopez · 2026-02-26T14:02:50Z

And now all the tests passed. There is some instability in F44, but not related to this PR.

alexey-tikhonov · 2026-02-26T14:59:13Z

And now all the tests passed. There is some instability in F44, but not related to this PR.

It is very suspicious that it was test_infopipe__list_by_name that I didn't see failing before.
Can there be a race condition in the test itself that is triggered by slow runner?

aplopez · 2026-02-26T16:22:53Z

It is very suspicious that it was test_infopipe__list_by_name that I didn't see failing before. Can there be a race condition in the test itself that is triggered by slow runner?

I thought the same until I noticed this test failed once and never again. The second time a completely different test failed. The third time, the latest, none.

gemini-code-assist bot reviewed Jan 21, 2026

View reviewed changes

src/db/sysdb_search.c Show resolved Hide resolved

alexey-tikhonov self-assigned this Jan 21, 2026

alexey-tikhonov self-requested a review January 21, 2026 14:44

aplopez force-pushed the enumerate branch from 955232a to f98eca5 Compare January 21, 2026 18:05

alexey-tikhonov reviewed Jan 22, 2026

View reviewed changes

src/db/sysdb_search.c Show resolved Hide resolved

aplopez added the backport-to-sssd-2-9 label Jan 22, 2026

alexey-tikhonov reviewed Jan 22, 2026

View reviewed changes

src/db/sysdb_search.c Outdated Show resolved Hide resolved

alexey-tikhonov added the Bugzilla label Jan 22, 2026

alexey-tikhonov requested a review from sumit-bose January 22, 2026 13:45

alexey-tikhonov assigned sumit-bose Jan 22, 2026

aplopez force-pushed the enumerate branch from f98eca5 to c78e8f6 Compare January 22, 2026 15:47

aplopez force-pushed the enumerate branch from c78e8f6 to 631e4be Compare January 23, 2026 10:05

alexey-tikhonov reviewed Feb 12, 2026

View reviewed changes

aplopez marked this pull request as ready for review February 24, 2026 13:19

aplopez added the Waiting for review label Feb 24, 2026

aplopez added 6 commits February 24, 2026 17:52

SYSDB: Remove unused function

efcd2a1

Function sysdb_enumpwent() is not used. It was replaced by sysdb_enumpwent_filter().

NSS: Reduce a possibly extremely long log message

4ba7fa8

When there are too many users (17,000+) this message can be too long. Limit it to the first 50 characters. Resolves: SSSD#6951

NSS: Fix wrong condition invalidating an optimization

e75895b

We must look into the TS cache only when a name is provided. Using the TS cache on an unfiltered enumeration is useless. Resolves: SSSD#6951

TESTS: Improve test_sysdb_enumpwent_filter

ccff788

Added a case that was not checked before. It is the case when `attr`, `attr_name` and `addtl_filter` are all `NULL`.

NSS: Some optimizations.

51416a4

Create the filter to retrieve only the requested entries. Do not create a new filter and search for matches if there is no results from the previous search. The called functions handle this case correctly but why waisting time calling them?

NSS: Be coherent when using a lastUpdate filter

ed557b2

Function cache_req_user_by_filter_lookup() will set or not the recent filter depending on whether data->name.attr is set or not. As mentioned in the comment, it should be done base on whether the refernced attribute is name or not.

aplopez force-pushed the enumerate branch from b3819b1 to ed557b2 Compare February 24, 2026 16:54

alexey-tikhonov added the coverity Trigger a coverity scan label Feb 24, 2026

alexey-tikhonov removed the coverity Trigger a coverity scan label Feb 24, 2026

Conversation

aplopez commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexey-tikhonov commented Jan 22, 2026

Uh oh!

aplopez commented Jan 22, 2026

Uh oh!

alexey-tikhonov commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-tikhonov Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

aplopez Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

aplopez Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aplopez Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aplopez Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aplopez Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov commented Feb 24, 2026

Uh oh!

alexey-tikhonov commented Feb 25, 2026

Uh oh!

aplopez commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aplopez commented Feb 26, 2026

Uh oh!

alexey-tikhonov commented Feb 26, 2026

Uh oh!

aplopez commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aplopez commented Jan 21, 2026 •

edited

Loading

alexey-tikhonov commented Jan 23, 2026 •

edited

Loading

alexey-tikhonov Feb 24, 2026 •

edited

Loading

aplopez Feb 24, 2026 •

edited

Loading

alexey-tikhonov Feb 24, 2026 •

edited

Loading

aplopez commented Feb 25, 2026 •

edited

Loading

aplopez commented Feb 26, 2026 •

edited

Loading