Several input sources at once `-f` `-uf` `-ad`

[spirillen]

Is your feature request related to a problem? Please describe.
As we are working toward an central Database with the API connector and several slave instances for serving the test loop, we will be missing the data feeders for what to test.

This can be partially fixed by allowing more than one -f -uf -ad pr @PyFunceble initiation.

Describe the solution you'd like
If we implant a 1 to many -f -uf -ad we can now change our sources test string to obtain the data over the internet protocol :// and thereby keep a better track of the sources pyfunceble_file and keep that as little as possible, and it will help keeping track of same sources.

IE. submit_here/wildcard.list is equal https://raw.githubusercontent.com/mypdns/porn-domains/master/submit_here/wildcard.list

What currently is happening, is that I can only think of one solution to this.

https://github.com/mypdns/porn-domains/blob/aefae394d0b8ea81c824a2c7d5d2cc29cf1442f4/travis/PrepareData.sh#L33-L70

This is combining a lot of lists, into one for testing, and it is actually ruing the real source and classification of a given record, the second issue to this is that the URI I'm using, would be from a previous commit and not the current one: Retesting old data, and only because the records can't be tested within one cycles I actually gets the new contribution tested by the second cycle.

https://github.com/mypdns/porn-domains/blob/d27450cbd3b64b68e78e088bd49f121e5ab2f778/travis/DataTesting.sh#L5-L20

Describe alternatives you've considered
A hole new program that can feed the PyFunceble database, but it would have to become either tricked by an external source (API) or monitoring a hole bunch of sources for changes (Lots of look ups)

Additional context
I'm not trying to say this is the solution for the above project, but it could be on the road to.

This would also enhance the way you can keep any CI/CD going, and yet testing several source files/URI by same code for other purposes than this example.

Challenge
Keep the the structure of input to the output where ie. hosts(in) = hosts(out) & wildcard(in) = wildcard(out) in projects not using a central DB with a separate data extraction tool for test results.

End note
I believe we should have an open debate on this topic as our combined projects has started taking of and yet there are many different usages of this program by many users

Invitations
I would like to invite at least @mitchellkrogza, @funilrys, @ZeroDot1, @ybreza and @shsingh to contribute to this suggestion and hear your feedback and maybe very very good idea's and suggestions.

[spirillen]

Appendix 1
It have crossed my mind, that it might be useful for some projects such as the https://github.com/mitchellkrogza/Phishing.Database to actually be able to mix the domain & url testing into one pyfunceble test string?

[funilrys]

@spirillen, the -ad is a bit special. And I won't implement it now. But the rest is implemented into 4.0.0a1 which will be available for general testing soon..

[funilrys]

Note to the people who are wondering, because of this issue the structure of the output directory changed.

Meaning that in the previous version it used to be output/domains/ACTIVE/list (for example). But in the new version, because we have to take into account that the end-user may give us multiple files to test, the path now becomes output/{file_name_without_special_chars}/domains/ACTIVE/list.

To summarise:
output/domains/ACTIVE/list is now output/{file_name_without_special_chars}/domains/ACTIVE/list.

Please be sure to update your automated script as soon as 4.0.0 will be out of alpha.

---

Note to self: Document this case.

[funilrys]

Also documented in the update section.

Closing. Feel free to test and reopen if not fixed.

[spirillen]

Please add to a re-test project.. CPU's still tied up

[funilrys]

@spirillen done.