container lint: option to not truncate

[vrothberg]

bootc container lint truncates the output, for instance, by only listing the first N paths as shown below. This is a perfectly sane default but I am seeking an option to list everything and not truncate. This would be especially helpful for running automated analyses and reports (which I am doing at the moment).

Lint warning: var-tmpfiles: Found content in /var missing systemd tmpfiles.d entries:
  d /var/cache/dnf/appstream-4d3c2aed03a3184f 0755 root root - -
  d /var/cache/dnf/appstream-4d3c2aed03a3184f/packages 0755 root root - -
  d /var/cache/dnf/appstream-4d3c2aed03a3184f/repodata 0755 root root - -
  d /var/cache/dnf/baseos-bea24aa3f40f8fdb 0755 root root - -
  d /var/cache/dnf/baseos-bea24aa3f40f8fdb/repodata 0755 root root - -
  ...and 7 more
Found non-directory/non-symlink files in /var:
  var/lib/dnf/history.sqlite
  var/lib/dnf/history.sqlite-shm
  var/lib/dnf/history.sqlite-wal
  var/lib/dnf/repos/appstream-4d3c2aed03a3184f/countme
  var/lib/dnf/repos/baseos-bea24aa3f40f8fdb/countme
  ...and 12 more

[erikh]

I looked into this preparing to make this patch, and it appears that https://github.com/bootc-dev/bootc/blob/main/utils/src/iterators.rs#L17 is the culprit and happens in a few spots. Perhaps a -v option to pass a negative integer or 0 for max could be used to instruct this routine to return all entries? Option<usize> might be more appropriate. Wanted to ask as it will take some time to build out a dev environment.

[cgwalters]

Perhaps a -v option to pass a negative integer or 0 for max could be used to instruct this routine to return all entries?

Sure, though in practice that's equivalent to simply not filtering the iterator at all i.e. not calling the function 😄

I think we can fix two problems at once here though...there's also duplication here with the "print the results" logic, and so I think if we had a wrapper for this in the cli code it could also handle the "full results" logic.

This all said...we didn't define here what the "option to not truncate" looks like. What I'd actually strawman instead is we have a --logfile option that is always verbose. For example in a CI/CD system I could do podman build -v $(pwd)/logs:/run/logs ... and then RUN bootc container lint --logfile /run/logs/bootc-lint.txt or so, and then use tooling like GitHub Actions artifacts (and all the equivalent) to attach these log artifacts to the run.

If we talk about writing a log file then of course we probably want to think too about outputting structured data (maybe YAML?)...

[M-Pixel]

Actually, I disagree with the statement "This is a perfectly sane default". All or nothing are the only reasonable defaults.

A particular dnf install command I ran outputs

Dependencies resolved.
======================================================================================================
 Package                                     Arch    Version                          Repo        Size
======================================================================================================
Upgrading:
 avahi-libs                                  x86_64  0.8-22.el9                       baseos      68 k
 glibc                                       x86_64  2.34-192.el9                     baseos     2.0 M
 glibc-common                                x86_64  2.34-192.el9                     baseos     304 k
 glibc-gconv-extra                           x86_64  2.34-192.el9                     baseos     1.7 M
 glibc-minimal-langpack                      x86_64  2.34-192.el9                     baseos      20 k
 libipa_hbac                                 x86_64  2.9.6-4.el9.2                    baseos      35 k
 libnvme                                     x86_64  1.13-1.el9                       baseos     111 k

followed by several more packages. Would any of us suggest that dnf would be improved by truncating that to

Dependencies resolved.
======================================================================================================
 Package                                     Arch    Version                          Repo        Size
======================================================================================================
Upgrading:
 avahi-libs                                  x86_64  0.8-22.el9                       baseos      68 k
 glibc                                       x86_64  2.34-192.el9                     baseos     2.0 M
 glibc-common                                x86_64  2.34-192.el9                     baseos     304 k
 glibc-gconv-extra                           x86_64  2.34-192.el9                     baseos     1.7 M
 glibc-minimal-langpack                      x86_64  2.34-192.el9                     baseos      20 k
 ... and 20 others

Actually, in the case of an often manual user command like dnf, it might be desirable for the user to not fill their terminal history with the details. But for that we have --quiet.

A lint is a lint. The purpose is to automatically find all problems of a particular type. Actually reporting everything that it finds is not "verbose", it's the norm. "Verbose" would be, for example, printing the line number within passwd of users reported by the sysusers lint.

If I think outside of the box, I can imagine a scenario or two where one would ant to run the lint but not "waste" tty space on the full output (just get the return value - did it pass or not?). In this scenario, even the truncated output is undesirable.

In typical container workflows, what is the use case for a truncated output? I see there's a problem, I get some of the details. So I fix those, re-run the containerfile, and repeat that process (which may involve installing hundreds of packages), which is unnecessarily convoluted; or I enter the container and use a command to get the full list of relevant files, which obviates the utility of the lint output, and is not convenient in any iteration loop that involves orchestration (and probably requires re-running the step at least 1 more time). Putting the full output in a log file is also not convenient in most Containerfile workflows, where failed steps leave the developer with nothing more than log output to diagnose what went wrong (getting a running shell in a container running from the post-failure state requires extra steps).

After all, it appears that this is not an intentional design decision, so much as a convenient implementation decision stemming from the iterator design.

[cgwalters]

After all, it appears that this is not an intentional design decision, so much as a convenient implementation decision stemming from the iterator design.

No it is intentional, we had to go to some effort to trim it. The rationale for the status quo is that - especially with the /var lint - not everything there is a big problem, and sometimes there can be a lot of content there, and it could really bloat logs.

I totally get your perspective though.

I think the most useful thing for us to do here though is #216 which would greatly help with the biggest source for lint spam from the var-tmpfiles.