[AIRFLOW-5834] Option to skip serve_logs process with workers

[msumit]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira
  • https://issues.apache.org/jira/browse/AIRFLOW-5834

Description

• Provide an option to skip serve_logs process along with workers

Tests

• Added

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

[feluelle]

@msumit master is currently failing. A potential fix #6486 is already there. Therefore I cancelled your tests. Can you do a rebase after the PR is merged, please? :)

[mik-laj]

I am not sure if this is the best solution. We will add an additional switch, but the community will be low when we should use it. This server is only needed when using FileTaskHandler. Do you think it is possible to make this server start only when this logger is selected?

[msumit]

Thanks @feluelle.

@mik-laj agrees that it'll add another hook to the system. However, I'm not sure if starting a log server process for FileTasKHandler is correct, as I was using that only and writing logs to a persistent volume, and don't need this process at all.

IMO we should not start this process along with workers and let folks decide if they really need it or not?

[OmerJog]

@msumit can you take a look at the test you added?

test_serve_logs_on_worker_start (tests.cli.test_cli.TestCLI) ... WARNI [airflow.utils.log.logging_mixin.LoggingMixin] Starting serve logs process within worker is going to be deprecated, and will be removed from Airflow 2.0
Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!
If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).

[msumit]

@OmerJog as of now I'm not able to set that var C_FORCE_ROOT while invoking the worker command. Had asked the same in Slack as well, but got no response so far.