BUG | `--config-dir`

[spirillen]

What is the problem you are experiencing?

I am experiencing a problem when I'm using the --config-dir option. From the appointed dir /home/$USER/Projects/github/funilrys/PyFunceble/.PyFunceble/ && ./.PyFunceble/ (tested both) I can't see all the values are obeyed, In Example the

# Activates the output of all information.
  all: yes

I would expect to see all the test rows, but I do only see the standard rows like Status and Source, It might be worth mentioning that I do have the same settings within my standard ~/.config/PyFunceble For new users, this is documented here https://docs.pyfunceble.com/use/configuration/location.html#linux-macos

How can we reproduce the problem?

To reproduce the problem:

1. Download repo and activate the VENV
2. Run pyfunceble --config-file ./.PyFunceble/.PyFunceble.overwrite.yaml --config-dir ./.PyFunceble/ -f /home/spirillen/Projects/github/mypdns/matrix/source/spyware/domains.list -f /home/spirillen/Projects/github/mypdns/matrix/source/spyware/wildcard.list -f /tmp/spyware.tsv

Do you have a screenshot?

Screenshot

What did you expect to happen?

That all my setting from the files of the config dir to be read and used

tree -af --prune .PyFunceble
.PyFunceble
├── .PyFunceble/iana-domains-db.json
├── .PyFunceble/inactive.csv
├── .PyFunceble/public-suffix.json
├── .PyFunceble/.pyfunceble-env
├── .PyFunceble/.pyfunceble_intern_downtime.json
├── .PyFunceble/.PyFunceble.overwrite.yaml
├── .PyFunceble/.PyFunceble.yaml
├── .PyFunceble/user_agents.json
└── .PyFunceble/whois.csv

Is there a workaround?

It seems you have specify the config file explicitly

--config-file /home/$USER/Projects/github/funilrys/PyFunceble/.PyFunceble/.PyFunceble.overwrite.yaml

How did you configure PyFunceble?

debug:
  # Provides everything related to the debug mode.

  # Activates the debug/logging.
  active: no
  # Sets the logging level.
  level: info

cli_decoding:
  # Provides everything related to the decoding from the CLI.

  # Activates the aggressive decoding.
  aggressive: no

  # Activates the decoding using the adblock decoder.
  adblock: no

  # Activates the decoding using the RPZ decoder.
  rpz: no

  # Activated the decoding using the Wildcard decoder.
  wildcard: no

cli_testing:
  # Provides everything directly related to the CLI testing.

  # Sets the Hosts IP to use while generation the hosts file(s)
  hosts_ip: '0.0.0.0'

  # Sets the number of maximal workers to use.
  # If set to null, the system use: CPU * Cores - 2
  max_workers: null

  # Activates the automatic continuation after a break or shortage.
  autocontinue: yes

  # Activates the storage and creation of a database of all INACTIVA and INVALID
  # subjects for continuous testing.
  inactive_db: yes

  # Activates the storage of some WHOIS information in order to avoid having
  # to spam the WHOIS servers.
  whois_db: yes

  # Activates the generation of complements.
  complements: no

  # Activates the expansion of CIDR to single addresses.
  cidr_expand: no

  # Sets the cooldown time to apply between each tests.
  cooldown_time: 0.0

  # Sets the Database Connector type to use.
  # Available: csv | mariadb | mysql | postgresql
  db_type: postgresql

  # Sets the filter to apply while reading the given input.
  # For example, if you give `\.info`, we will only test the subjects who match
  # `\.info`.
  file_filter: null

  # Activates the mining of data.
  mining: no

  # Acknowledges that we may test for local network component.
  # NOTE: Activating this, will remove the syntax checker completely.
  local_network: no

  # Activates or disable the preloading of the given input file(s).
  # When this is activated, we take the time to pre load and decode the content
  # of your file so that we can optimize some of our process regarding the
  # auto continue.
  #
  # Note: This has no effect if the auto-continue is deactivated.
  preload_file: yes

  # Activates or disables a chancy tester mode that unleashes the safety
  # workflow in place.
  #
  # WARNING:
  #   You shouldn't have to use this unless you feel really lucky and trust your
  #   machine.
  #
  #   This mode makes things look 'fast', but it may produce some unexpected
  #   results if :code:`N` process simultaneously write the same output file.
  #
  #   This mode makes the graphical CLI output unparsable - either.
  #
  #   **MAY THE FORCE BE WITH YOU!**
  chancy_tester: no

  ci:
    # Provides everything related to the Continuous integration.

    # Activates the continuous integration mode.
    # WARNING: Do not activate without asking or knowing what you are doing.
    active: no

    # Sets the commit message to apply everytime except for the last one.
    commit_message: 'PyFunceble - AutoSave'

    # Sets the commit message to apply at the very end of the test.
    end_commit_message: 'PyFunceble - Results'

    # Sets the number of minutes  to wait before starting to save and stop
    # a session.
    max_exec_minutes: 15

    # Sets the working branch. This is the branch from where we are testing.
    branch: master

    # Sets the distribution branch. This is the branch that is going to get
    # the (final) results.
    distribution_branch: master

    # Sets the command to execute everytime before each commits except the last
    # one.
    command: null

    # Sets the command to execute before the last commit.
    end_command: null

  display_mode:
    # Provides everything related to the display mode.

    # Activates the printing of dots.
    dots: no

    # Activates the printing of the execution time.
    execution_time: yes

    # Activates the output of the percentage information.
    percentage: yes

    # Activates the output of the top registrar information.
    registrar: no

    # Activates the quiet mode.
    quiet: no

    # Activates the output of very few information.
    less: no

    # Activates the output of all information.
    all: yes

    # Activates the output of the status and it's status (only)
    simple: no

    # Activates the coloration of stdout.
    colour: yes

    # Sets the status do display to STDOUT.
    # NOTE: This does not have any effect with the generated files.
    # Available values: ALL | ACTIVE | INACTIVE | INVALID | SANE | MALICIOUS
    status: all

    # Sets the maximum number of registrar to display.
    # NOTE: This does not have any effect with the generated files.
    max_registrar: 0

  testing_mode:
    # Provides all available testing modes.
    # WARNING: Only one can be used at a time.

    # Activates the availability test.
    availability: yes

    # Activates the syntax test.
    syntax: no

    # Activates the reputation test.
    reputation: no

    # BETA: Activates the platform contribution test.
    platform_contribution: yes

  days_between:
    # Provides everything which is x days periodic.

    # NOT IMPLEMENTED (Anticipation for future usage).
    db_clean: 28

    # Provides the number of days to wait before retesting subject which were
    # stored into the inactive database.
    db_retest: 7

  sorting_mode:
    # Provides all our sorting mode.
    # WARNING: The sorting mode is only applied to the files. NOT STDOUT.

    # Activates the hierarchical sorting.
    hierarchical: yes

    # Activates the standard sorting.
    standard: no

  file_generation:
    # Provides everything related to the generation of file.

    # Deactivates the generation of any status file.
    no_file: no

    # Activates the generation of the hosts file(s).
    hosts: no

    # Activates the generation of the plain (or raw) file(s).
    plain: yes

    # Activates the generation of the analytic file(s).
    analytic: no

    # Activate the generation of an unified file for the copy of what is
    # outputted to STDOUT.
    unified_results: no

    # Activates or disables the merging of the results of all inputted files into
    # one single output subdirectory.
    merge_output_dirs: no

lookup:
  # Provides everything related to the lookup.

  # Activates the usage of the DNS lookup.
  dns: yes

  # Activates the usage of the HTTP status code lookup.
  http_status_code: yes

  # Activates the usage of the network information socket.
  netinfo: yes

  # Activates the usage of our special rules.
  special: yes

  # Activates the usage of the WHOIS record.
  whois: yes

  # Activates the usage of the reputation data reputation.
  reputation: no

  # Activate the usage of the platform to lookup status.
  platform: yes

  # Sets the timeout to apply to each of our lookup tools who support a timeout
  # option.
  timeout: 5

dns:
  # Provides everything related to the DNS lookup.

  # Activates the follow-up of the given order.
  follow_server_order: no

  # Activates the trust mode. Meaning that if one of the DNS give us a negative
  # response (without error), we take it as it is.
  # When this mode is deactivated (no), when one of the DNS give us a negative
  # response (without error), we still ask all other servers.
  trust_server: yes

  # Sets the list of server to communicate with.
  # It must be a list.
  #
  # WARNING:
  #   IPv6 should be given in this format if a port is explicitly given:
  #     [ip]:port
  #   If you omit the braked, the port will be set to the default one (53).
  #
  # As example:
  #   - first.dns
  #   - second.dns
  #
  server:
    # - 192.168.1.6
    # - 192.168.1.105
    - 9.9.9.10
    - 149.112.112.10

  # Sets the protocol to use.
  # Available: UDP | TCP | HTTPS | TLS
  protocol: UDP

  # Sets the delay (seconds) to apply between each query.
  # When a value > 0.00000 is given, we will wait for the given amount of
  # seconds between each query.
  delay: 0.0

# Not Implemented yet. Reserved for future usage and implementation.
share_logs: no

user_agent:
  # Provides everything related to the user agent choice.

  browser: firefox
  platform: linux

  # Sets the user agent to use.
  # WARNING: If given, this will be used systematically.
  custom: null

proxy:
  # Provides everything related to the proxy.
  #
  # The idea:
  #   We have two main keys, "global" and "rules".
  #   The system will always follow the global keys unless you define an explit
  #   TLD.
  #
  # Example:
  #
  #   Let's say we want all HTTP requests to go through example.org but we want
  #   all HTTP request for domains ending with `.com`, `.org` and `.dev` to go
  #   through example.com. And, we want all domains ending with `.onion` to go
  #   through example.dev.
  #
  #   This is how it will look like.
  #
  #       global:
  #         http: http://example.org:8080
  #         https: http://example.org:8080
  #
  #       rules:
  #         - http: http://example.com:8080
  #           https: http://example.org:8080
  #           tld:
  #             - com
  #             - org
  #             - dev
  #         - http: socks5h://example.dev:8080
  #           https: socks5h://example.dev:8080
  #           tld:
  #             - onion
  #
  rules:
    - http: socks5h://nfs.matrix.lan:9050
      https: socks5h://nfs.matrix.lan:9050
      tld:
        - onion
  # global:
  #   http: socks5h://nfs.matrix.lan:9050 #http://nfs.matrix.lan:3128
  #   https: socks5h://nfs.matrix.lan:9050

# Activates the verification of the certificate.
verify_ssl_certificate: no

# Set the maximum number of retries to perform.
# This should be an integer >= 0.
max_http_retries: 2

http_codes:
  # Provides everything related to the HTTP code lookup interpretation.

  # Informs PyFunceble that the end-user (not PyFunceble) is managing this list.
  # This means that you won't get any update in this section.
  self_managed: yes

  list:
    up:
      - 100
      - 101
      - 102
      - 200
      - 201
      - 202
      - 203
      - 204
      - 205
      - 206
      - 207
      - 208
      - 226
      - 429
    potentially_down:
      - 400
      - 402
      - 409
      - 410
      - 412
      - 414
      - 415
      - 416
      - 451
    potentially_up:
      - 000
      - 300
      - 301
      - 302
      - 303
      - 304
      - 305
      - 307
      - 308
      - 403
      - 404
      - 405
      - 406
      - 407
      - 408
      - 411
      - 413
      - 417
      - 418
      - 421
      - 422
      - 423
      - 424
      - 426
      - 428
      - 431
      - 500
      - 501
      - 502
      - 503
      - 504
      - 505
      - 506
      - 507
      - 508
      - 510
      - 511

links:
  # Provides some of our API links.
  # NOTE: Not implemented yet as the underlying infrastructure will be
  # rewritten.
  api_date_format: 'https://pyfunceble.funilrys.com/api/date-format'
  api_no_referrer: 'https://pyfunceble.funilrys.com/api/no-referrer'

platform:
  # Provides everything related to the platform.

  # Activates the push of dataset into the platform.
  # WARNING: This is useless, if you don't have an API Token set as the
  # PYFUNCEBLE_PLATFORM_API_TOKEN environment variable.
  push: yes

  # Sets the preferred pull origin.
  #
  # When choosing `recommended`, the system will pull the recommended status
  # that is provided and calculated by the platform.
  #
  # When choosing, `frequent`, the system will pull the most frequent status
  # that is provided and calculated by the platform.
  #
  # When choosing, `latest`, the system will pull the latest status that is
  # provided and available into the platform.
  #
  # Available: frequent | latest | recommended
  preferred_status_origin: recommended

  # Defines the checker type to prioritize when trying to fullfil platform
  # contracts.
  #
  # Notes:
  #   1. This is a list. The order matters.
  #   2. One can overwrite this value, by settings a comma separated list of
  #      checker type to prioritize through the PYFUNCEBLE_PLATFORM_CHECKER_PRIORITY
  #      environment variable.
  #   3. When set to `none`, the platform will throw a random contract at us.
  #
  # Example:
  #   Prioritize availability checker until no contract is available, then
  #   prioritize syntax checker until no contract is available, then prioritize
  #   reputation checker until no contract is available.
  #
  #   - availability
  #   - syntax
  #   - reputation
  #
  # Available: none | availability | reputation | syntax
  checker_priority:
    - availability
    - syntax

  # Defines the checker type to exclude when trying to fullfil platform
  # contracts.
  #
  # Notes:
  #   1. This is a list.
  #   2. One can overwrite this value, by settings a comma separated list of
  #      checker type to exclude through the PYFUNCEBLE_PLATFORM_CHECKER_EXCLUDE
  #      environment variable.
  #   3. When set to `none`, no checker type will be excluded.
  #
  # Example:
  #   Exclude the reputation checker from the list of checker to use to fullfil.
  #
  #   - reputation
  #
  # Available: none | availability | reputation | syntax
  checker_exclude:
    - reputation

Which Operating System did you use?

Ubuntu 24.04 LTS

Which Python Version did you use?

3.12

Which PyFunceble Version did you use?

4.3.0a4.dev (Blue Duckling: Tulip)

Additional context

The PostgreSQL module are now working as supposed 👍🏻

[spirillen]

The only way I can get the display to work is by adding this as a string to the test line OR follow "Is there a workaround?"

---

Oh, the PYFUNCEBLE_OUTPUT_LOCATION are not obeyed either...

Configured value: PYFUNCEBLE_OUTPUT_LOCATION=./.PyFunceble

---

A third observation is, that only the last of the files set to be tested, seems to actually being tested... (Have not checked the actual contents of these result files, but the total number of lines seems to match)

Actual value: output/

Tree output

tree -af --prune output/
output
└── output/spyware.tsv
    ├── output/spyware.tsv/Analytic
    │   ├── output/spyware.tsv/Analytic/ACTIVE
    │   │   └── output/spyware.tsv/Analytic/ACTIVE/.gitkeep
    │   ├── output/spyware.tsv/Analytic/.gitkeep
    │   ├── output/spyware.tsv/Analytic/POTENTIALLY_ACTIVE
    │   │   └── output/spyware.tsv/Analytic/POTENTIALLY_ACTIVE/.gitkeep
    │   ├── output/spyware.tsv/Analytic/POTENTIALLY_INACTIVE
    │   │   └── output/spyware.tsv/Analytic/POTENTIALLY_INACTIVE/.gitkeep
    │   └── output/spyware.tsv/Analytic/SUSPICIOUS
    │       └── output/spyware.tsv/Analytic/SUSPICIOUS/.gitkeep
    ├── output/spyware.tsv/domains
    │   ├── output/spyware.tsv/domains/ACTIVE
    │   │   └── output/spyware.tsv/domains/ACTIVE/.gitkeep
    │   ├── output/spyware.tsv/domains/.gitkeep
    │   ├── output/spyware.tsv/domains/INACTIVE
    │   │   └── output/spyware.tsv/domains/INACTIVE/.gitkeep
    │   ├── output/spyware.tsv/domains/INVALID
    │   │   └── output/spyware.tsv/domains/INVALID/.gitkeep
    │   ├── output/spyware.tsv/domains/MALICIOUS
    │   │   └── output/spyware.tsv/domains/MALICIOUS/.gitkeep
    │   ├── output/spyware.tsv/domains/SANE
    │   │   └── output/spyware.tsv/domains/SANE/.gitkeep
    │   └── output/spyware.tsv/domains/VALID
    │       └── output/spyware.tsv/domains/VALID/.gitkeep
    ├── output/spyware.tsv/hosts
    │   ├── output/spyware.tsv/hosts/ACTIVE
    │   │   └── output/spyware.tsv/hosts/ACTIVE/.gitkeep
    │   ├── output/spyware.tsv/hosts/.gitkeep
    │   ├── output/spyware.tsv/hosts/INACTIVE
    │   │   └── output/spyware.tsv/hosts/INACTIVE/.gitkeep
    │   ├── output/spyware.tsv/hosts/INVALID
    │   │   └── output/spyware.tsv/hosts/INVALID/.gitkeep
    │   ├── output/spyware.tsv/hosts/MALICIOUS
    │   │   └── output/spyware.tsv/hosts/MALICIOUS/.gitkeep
    │   ├── output/spyware.tsv/hosts/SANE
    │   │   └── output/spyware.tsv/hosts/SANE/.gitkeep
    │   └── output/spyware.tsv/hosts/VALID
    │       └── output/spyware.tsv/hosts/VALID/.gitkeep
    ├── output/spyware.tsv/ips
    │   ├── output/spyware.tsv/ips/ACTIVE
    │   │   └── output/spyware.tsv/ips/ACTIVE/.gitkeep
    │   ├── output/spyware.tsv/ips/.gitkeep
    │   ├── output/spyware.tsv/ips/INACTIVE
    │   │   └── output/spyware.tsv/ips/INACTIVE/.gitkeep
    │   ├── output/spyware.tsv/ips/INVALID
    │   │   └── output/spyware.tsv/ips/INVALID/.gitkeep
    │   ├── output/spyware.tsv/ips/MALICIOUS
    │   │   └── output/spyware.tsv/ips/MALICIOUS/.gitkeep
    │   ├── output/spyware.tsv/ips/SANE
    │   │   └── output/spyware.tsv/ips/SANE/.gitkeep
    │   └── output/spyware.tsv/ips/VALID
    │       └── output/spyware.tsv/ips/VALID/.gitkeep
    ├── output/spyware.tsv/logs
    │   ├── output/spyware.tsv/logs/.gitkeep
    │   └── output/spyware.tsv/logs/percentage
    │       └── output/spyware.tsv/logs/percentage/.gitkeep
    ├── output/spyware.tsv/.running
    └── output/spyware.tsv/splitted
        └── output/spyware.tsv/splitted/.gitkeep

31 directories, 30 files

[funilrys]

PYFUNCEBLE_OUTPUT_LOCATION -> That's the expected behavior @spirillen. What is wrong with that ?

[spirillen]

PYFUNCEBLE_OUTPUT_LOCATION -> That's the expected behavior @spirillen. What is wrong with that ?

Most admit I no longer remember this case, yet from what I have been writing, I'll suspect the PYFUNCEBLE_OUTPUT_LOCATION wasn't followed and a default location was used for the output 🤷🏻‍♂️

[funilrys]

I worried about that, but I couldn't reproduce in my tests with past and incoming (pre)releases.