Squashed commit of the following:

commit 0582a27
Author: funilrys <[email protected]>
Date:   Sat Jul 16 17:50:29 2022 +0200

    Bump version to v4.1.2.dev

commit 1ddc3a0
Author: funilrys <[email protected]>
Date:   Sat Jul 16 14:38:45 2022 +0200

    Fix issue with encoding.

    This patch fixes #300.

    Indeed, before this patch, we were using the system-wide encoding
    instead of the defacto default UTF-8.

    Contributors:
      * @avatartw

commit 9a5fada
Author: funilrys <[email protected]>
Date:   Mon Mar 21 19:40:47 2022 +0100

    Bump version to v4.1.1

commit 07ad55f
Author: funilrys <[email protected]>
Date:   Mon Mar 21 19:35:53 2022 +0100

    Squashed commit of the following:

    commit 8320b72
    Author: funilrys <[email protected]>
    Date:   Mon Mar 21 19:29:53 2022 +0100

        Upgrade contributor list.

        This patch adds @bigdargon to the list of contributors.

    commit a448673
    Author: funilrys <[email protected]>
    Date:   Mon Mar 21 19:24:48 2022 +0100

        Better handling of subjects without extension.

        This patch fixes #297.
        This patch fixes dead-hosts/dev-center#23.

        This patch touches the proxy matcher.

        Indeed, before this patch, I didn't assume that it may be possible
        to have subject without any extension. This error lead some
        session to crash shortly before the end.

        This patch fixes the issue by handling the case that the subject
        without any extension is given.

        Contributors:
          * @spirillen
          * @bigdargon

commit a6f8b9c
Author: funilrys <[email protected]>
Date:   Sat Mar 12 12:41:30 2022 +0100

    Bump version to v4.1.0

commit 476a942
Author: funilrys <[email protected]>
Date:   Sat Mar 12 09:13:37 2022 +0100

    Bump version to v4.1.0b18.

commit 5cae08e
Author: funilrys <[email protected]>
Date:   Sat Mar 12 09:06:38 2022 +0100

    Restrict python-box.

    This patch fixes #294.

    Contributors:
      * @cdgriffith

commit 3688974
Author: funilrys <[email protected]>
Date:   Sun Mar 6 16:55:12 2022 +0100

    Bump version to v4.1.0b17

commit 7770336
Merge: 47a4e4b 06398ca
Author: funilrys <[email protected]>
Date:   Sun Mar 6 16:51:56 2022 +0100

    Merge branch 'maintenance' into dev

commit 06398ca
Author: funilrys <[email protected]>
Date:   Sun Mar 6 16:51:19 2022 +0100

    Add missing variable.

commit 47a4e4b
Author: funilrys <[email protected]>
Date:   Sun Mar 6 14:41:01 2022 +0100

    Bump version to v4.1.0b16.

commit 513a61f
Author: funilrys <[email protected]>
Date:   Sun Mar 6 14:22:26 2022 +0100

    Better handling of proxy usage.

    This patch touches #132.
    This patch fixes #293.

    Indeed, before this patch a special case related to the test on
    onions subject wasn't handled properly.

    This patch also replace the socks5 with socks5h protocol within
    the documentation. User that are testing onions and do not have
    a DNS resolver/server that can handle them are advised to use the
    socks5h protocol instead of socks5. That will ensure that the DNS
    lookup is not handled locally.

    Contributors:
      * @spirillen

commit 7a52279
Author: funilrys <[email protected]>
Date:   Sat Mar 5 23:55:19 2022 +0100

    Bump version to v4.1.0b15.

commit d721eef
Author: funilrys <[email protected]>
Date:   Sat Mar 5 23:53:14 2022 +0100

    fixup! Add sock capabilities.

commit 2f2eddb
Author: funilrys <[email protected]>
Date:   Sat Mar 5 21:53:33 2022 +0100

    Bump version to v4.1.0b14.

commit 03f9446
Author: funilrys <[email protected]>
Date:   Sat Mar 5 21:47:48 2022 +0100

    Add sock capabilities.

commit fdb9a27
Author: funilrys <[email protected]>
Date:   Sat Mar 5 19:50:13 2022 +0100

    Bump version to v4.1.0b13.

commit 5b813d7
Author: funilrys <[email protected]>
Date:   Sat Mar 5 19:45:29 2022 +0100

    Introduction of the *.altervista.org SPECIAL rule.

    This patch fixes #292.

    From now on, any subjects matching *.altervista.org that return
    the 403 status codes are supplied as `INACTIVE`.

    Contributors:
      * @spirillen

commit d73c097
Author: funilrys <[email protected]>
Date:   Sat Mar 5 19:41:46 2022 +0100

    Fix deployment to AUR.

commit 824afd9
Author: funilrys <[email protected]>
Date:   Sat Mar 5 19:23:51 2022 +0100

    Add undocumented SPECIAL rules.

commit b393b85
Author: funilrys <[email protected]>
Date:   Sat Mar 5 19:20:17 2022 +0100

    Update Copyright year.

commit 0fc29e6
Author: funilrys <[email protected]>
Date:   Sat Mar 5 18:54:38 2022 +0100

    Bump version to v4.1.0b12

commit 8738648
Author: funilrys <[email protected]>
Date:   Sat Mar 5 18:49:04 2022 +0100

    Introduction of the proxy settings.

    This patch fixes #132.

    Indeed, before this patch, one could use the HTTP_PROXY and
    HTTPS_PROXY settings. But, PyFunceble wouldn't actually
    handle some special rules.

    This patch introduce the `--http-proxy` and `https-proxy`
    arguments.

    This patch introduce the ability to define some extra rules -
    within the configuration file. Indeed, this is something that
    was missing.
    Therefore, from now on, you can define some extra rules around
    your proxies.

    For example, if you want to use `example.com` as main proxy but
    want all `.com` and `.org` subjects to be requested through
    `example.dev`, this is now possible through the configuration:

    ```yaml
    proxy:
      global:
        http: http://example.com:8080
        https: http://example.com:8080
      rules:
        - http: http://example.dev:8080
          https: http://example.dev:8080
          tld:
            - com
            - org
    ```

    Also:
      * Fix Codeblock syntax in the documentation.

    Contributors:
      * @spirillen

commit 152dc4e
Author: funilrys <[email protected]>
Date:   Sun Feb 13 18:53:58 2022 +0100

    Bump version to v4.1.0b11.

commit 22d39cf
Author: funilrys <[email protected]>
Date:   Sun Feb 13 18:28:25 2022 +0100

    Add forgotten explicit encoding.

    This patch fixes #291.

    Indeed, before this patch there may have been some encoding
    collision because I forgot to put the encoding explicitely before
    opening a CSV file.

    Contributors:
      * @avatartw

commit ccbf6fe
Author: funilrys <[email protected]>
Date:   Sat Feb 12 13:07:03 2022 +0100

    Bump version to v4.1.0b10.

commit ae46748
Author: funilrys <[email protected]>
Date:   Sat Feb 12 13:01:48 2022 +0100

    Introduction of the '*-lookup-only' argument.

    This patch fixes #276.

    Indeed, before this patch, there way no way to tell - from the
    CLI - that we only want the method X or Z for the lookup.

    Contributors:
      * @spirillen

commit 8ebfde6
Author: funilrys <[email protected]>
Date:   Sat Jan 1 17:36:31 2022 +0100

    Bump version to v4.1.0b9.

commit ab8f5e7
Author: funilrys <[email protected]>
Date:   Sat Jan 1 17:30:58 2022 +0100

    Improvement of the Support of IPv6 DNS.

    Indeed, before this patch, the support was not complete.
    This patch complete the support by expecting the following format
    for any IPv6 with explicit address:

        [ip]:port

    If no braket is found, we will provide the default port.

commit 2640455
Author: funilrys <[email protected]>
Date:   Sat Jan 1 16:11:09 2022 +0100

    Bump version to v4.1.0b8.

commit 16f3898
Author: funilrys <[email protected]>
Date:   Sat Jan 1 16:07:07 2022 +0100

    fixup! Do DNS Lookup before querying URL.

commit b01376b
Author: funilrys <[email protected]>
Date:   Sat Jan 1 14:37:52 2022 +0100

    fixup! Fix issue with deployment to the AUR.

commit 2393ff3
Author: funilrys <[email protected]>
Date:   Sat Jan 1 13:47:32 2022 +0100

    Fix issue with deployment to the AUR.

commit 408e277
Author: funilrys <[email protected]>
Date:   Fri Dec 31 18:40:50 2021 +0100

    Bump version to v4.1.0b7.

commit cd14dc8
Author: funilrys <[email protected]>
Date:   Fri Dec 31 18:21:33 2021 +0100

    Do DNS Lookup before querying URL.

    This patch fixes #164.

    Indeed, before this patch, we were trying to query URL even if
    the domain didn't have an IP address.
    This patch fixes the issue by doing a classic (NS, CNAME, A, AAA)
    query before trying to query any URL.

    Note: The DNS Lookup is the same as the one we do for a standard
    domain/ip test.

    Contributors:
      * @mitchellkrogza
      * @spirillen

commit 86e2558
Author: spirillen <[email protected]>
Date:   Fri Dec 31 01:38:33 2021 +0100

    Slight change to the DNS IP

    Have changed the Quad9 IP address to 9.9.9.10 as we should promoted unfiltered DNS services
    In general, as that hopefully is the most needed in combination with @PyFunceble

commit 8505492
Author: funilrys <[email protected]>
Date:   Fri Dec 31 16:30:48 2021 +0100

    Bump version to v4.1.0b6.

commit 088947b
Author: funilrys <[email protected]>
Date:   Fri Dec 31 16:11:24 2021 +0100

    Workaround Y2K38.

    This patch fixes #261.

    Indeed, before this patch, there was no proper solution to the
    Y2K38 issue - within PyFunceble.

    This patch try to workaround the issue by rounding the dates to
    a few weeks before Y2K38 - when the database vendor didn't
    implement a solution.

    Contributors:
      * @spirillen

commit 2e2c517
Author: funilrys <[email protected]>
Date:   Sat Dec 25 22:32:25 2021 +0100

    Bump version to v4.1.0b5.

commit 2c4ea7d
Merge: b3c4442 8514ca5
Author: funilrys <[email protected]>
Date:   Sat Dec 25 22:28:20 2021 +0100

    Merge remote-tracking branch 'origin/maintenance' into dev

commit 8514ca5
Author: funilrys <[email protected]>
Date:   Sat Dec 25 20:29:11 2021 +0100

    Review Syntax checker.

    This patch fixes #282.

    Our testing module was too close to standards.
    If we consider blogspot.com a Public Suffix and therefore a effective
    Top-Level domain (eTLD), hello_world.blogspot.com shouldn't be valid.

    But the really behind the public suffixes is different.
    Therefore, this patch relaxes the rules regarding sub-domains behind a
    known public suffix.

commit a346154
Author: funilrys <[email protected]>
Date:   Sat Dec 25 19:09:14 2021 +0100

    Fix issue regarding false-positive invalid.

    This patch touches #282.

    Indeed, before this patch our self-healing false-positive lookup through
    DNS may have been incomplete.
    This patch fixes the issue by making the ruleset for the continuation to
    the next test method more transparent.

commit b3c4442
Merge: f0292cc f537119
Author: Nissar Chababy <[email protected]>
Date:   Sat Dec 25 17:28:34 2021 +0100

    Merge pull request #262 from veracioux:tuterm_demo

    Add automated CLI demonstrations

    Contributors:
      * @veracioux
      * @spirillen

commit f537119
Merge: f6c666e f0292cc
Author: funilrys <[email protected]>
Date:   Sat Dec 25 17:26:45 2021 +0100

    Merge remote-tracking branch 'upstream/dev' into tuterm_demo

commit f6c666e
Author: funilrys <[email protected]>
Date:   Sat Dec 25 16:56:52 2021 +0100

    Add @veracioux to the list of contributors.

commit f0292cc
Author: spirillen <[email protected]>
Date:   Sat Dec 25 16:45:03 2021 +0100

    Merge: #284

    This patch merges #284.
    This patch fixes #283.

    Contributors:
      * @spirillen

commit bb80fa5
Author: funilrys <[email protected]>
Date:   Wed Dec 22 19:06:15 2021 +0100

    Fix sphinx error.

commit 7b7ca75
Author: funilrys <[email protected]>
Date:   Wed Dec 22 18:59:02 2021 +0100

    Url with scheme and port shouldn't be INVALID.

commit ae9333a
Author: funilrys <[email protected]>
Date:   Wed Dec 22 17:19:04 2021 +0100

    Bump version to v4.1.0b4.

commit 869cab6
Merge: 6a66e71 d4d42bf
Author: funilrys <[email protected]>
Date:   Wed Dec 22 17:13:31 2021 +0100

    Merge remote-tracking branch 'origin/chancy-tester' into dev

commit 6a66e71
Merge: a8bea19 177a2f0
Author: funilrys <[email protected]>
Date:   Wed Dec 22 17:13:05 2021 +0100

    Merge remote-tracking branch 'origin/maintenance' into dev

commit d4d42bf
Author: funilrys <[email protected]>
Date:   Wed Dec 22 01:11:45 2021 +0100

    Introduction of the chancy tester mode.

    After discussion with some people and reading some emails of
    people privately making a comparison between PyFunceble 3 and 4, I
    decided to write the code for a "chancy" testing mode.

    Origin:
        The chancy mode is just a reviving of what I do not agree with
        - anymore.

        In PyFunceble 3, the test feedback is almost instant, meaning that
        the feeling of "speed" is there. But if you read the
        source code, you will see that it's possible because I chose to
        ignore a lot of safety to play with my luck, people's luck and
        machine's luck.

        Indeed, under PyFunceble 3, it is theoretically possible to have a
        "line overflow".

        That means that when:

            - You are using the multiprocessing mode
            - You are testing `example.org` and `example.net`,

        it may have been possible to get the following line as a result:

            examexampleple.org.net

        or this:

            example.orgexample.net

        That's the fact. If that rare case happens, you may have a
        corrupted output that you will later parse into your workflow.
        To solve the problem, I decided to change the workflow design.
        That's PyFunceble 4.

    Current State:

        Under PyFunceble 4, two main things changed:

            1. No more single and multiprocessing.
            2. No file production from the testing workers.

        What we used to call the `single-mode` went away and was
        replaced with a multiprocessing mode that's always there. The idea
        was that nowadays multiprocessing is the way.

        On the other hand, I killed the possible "overflow" of PyFunceble 3
        by following a clear data workflow that makes output safe.

        This is an approximate overview of the actual workflow:

        --- Read Subject ---> Tester Queue
                                 \
                                  ---> N Tester Process ---> Producer Queue
                                                            /
                                    1 Producer Process  <---
                                                 /
          Output File | Graphical CLI Output <---

        With this workflow, I'm sure that only ONE single process is
        reporting into any of the generated files. That means that the
        issue I previously described can't happen.

        That's the safe workflow. The downside of this workflow is that
        people think that everything is slow. It is. But for your safety.
        If you look at some very verbose logs, you will see
        that the actual testing is as quick as possible (for your hardware)
        but it's the file generation that is slow. That's because
        it goes one tested subject at a time - for your safety.

    Chancy Mode:

        The chancy mode brings back a PyFunceble 3 "like" way of handling
        data. That means that if used, no safety is given. An overflow can
        happen, but the feeling of "speed" is back.

        This mode exists but won't be actively maintained.

        To reduce long term maintenance of this mode, it has been built on
        top of the maintained and developed components of the workflow.
        Meaning that as long as no new component is built, this mode is
        "self-maintained".

commit 177a2f0
Author: funilrys <[email protected]>
Date:   Tue Dec 21 16:55:27 2021 +0100

    Add @gwarser to the list of contributors.

commit a8bea19
Author: funilrys <[email protected]>
Date:   Tue Dec 21 16:04:03 2021 +0100

    Bump version to v4.1.0b3.

commit 1806e10
Author: funilrys <[email protected]>
Date:   Tue Dec 21 15:38:39 2021 +0100

    Handle DNS name too long.

    Indeed before this patch, some DNS name which were longer than 255 octets
    may produce some fatal error.

commit 62d799c
Author: funilrys <[email protected]>
Date:   Sun Dec 19 12:11:15 2021 +0100

    fixup! Bump version to v4.1.0b2.

commit dad28cc
Merge: 1310461 b5ecdc2
Author: funilrys <[email protected]>
Date:   Sat Dec 18 22:59:43 2021 +0100

    Bump version to v4.1.0b2.

    Also:
      Fix #272.

commit 1310461
Author: funilrys <[email protected]>
Date:   Sat Dec 18 22:53:09 2021 +0100

    Bump version to v4.1.0b1.

commit 213a087
Author: funilrys <[email protected]>
Date:   Sat Dec 18 19:59:12 2021 +0100

    Add max internal max rescursion depth.

    Indeed, before this patch, the CNAME follower when doing HTTP request may
    habe been able to run without a limit.

    This patch fixes it by allowing a max depth of 60. Any depth above 60 will
    just be skipped.

commit ebc02c6
Author: funilrys <[email protected]>
Date:   Sat Dec 18 20:11:18 2021 +0100

    Introduction of a the max-http-retries arg.

    Indeed, before this patch, there was no way to control the maximum number
    of retries PyFunceble is allowed to perform when doing an HTTP request.

    This patch fixes that hole by adding the `--max-http-retries` argument.

commit fdc4f9c
Author: funilrys <[email protected]>
Date:   Sat Dec 18 22:31:46 2021 +0100

    Logical fix.

    Indeed, before this patch, we didn't actively checked if the inactive
    and autocontinue dataset  lookup was deactivated before doing some backup
    or update of the dataset.

    This patch fixes the issue by checking if we are authorized to process
    such dataset before even trying to do so.

commit d31b9b1
Author: funilrys <[email protected]>
Date:   Sat Dec 18 22:31:16 2021 +0100

    Fix issue with counter.

commit a85eafb
Author: funilrys <[email protected]>
Date:   Sat Dec 18 22:29:58 2021 +0100

    Review requester initialization.

    Indeed, before this patch, the way the requester was initialized wasn't
    safe enough for a multiprocessing environment.

commit c487a48
Author: funilrys <[email protected]>
Date:   Sat Dec 18 21:44:22 2021 +0100

    Add a way to differs worker startup and creation.

    Indeed, before this patch, as soon as something is put inside an input
    queue of a worker, the worker will be automatically created.
    But, at sometime, for example for worker that should only run at the end,
    we do not want that behavior.

    Therefore, this patch introduce a way to differ the creation and startup
    of a process - when necessary.
    This patch apply for example for the file sorter that is actually only
    necessary at the end of a test.

commit cb19031
Author: funilrys <[email protected]>
Date:   Sun Dec 5 19:05:36 2021 +0100

    Bump version ot v4.0.2.

commit b5ecdc2
Author: funilrys <[email protected]>
Date:   Sun Dec 5 18:18:30 2021 +0100

    Extract the registrar from the WHOIS record.

    This patch fixes #272.

    Indeed, before this patch, there was no way to get access to the registrar
    from the WHOIS record. This patch tries to solve the issue by adding a
    registrar extractor.

    This patch also introduce the `--registrar` and `--max-registrar` arguments
    that can be used to display the top N registrar that were extracted.
    Using the `--registrar` argument also generate a `logs/registrar.txt` file
    that can be used to list the all registrar found while testing the current
    file.

    Necessary Migration: -- All migration will be automatically triggered.
      * csv file: The whois.csv file needs to be migrated in order for it to
        have the `registrar` column.
      * mariadb: The whois_record tables need to be migrated in order for it
        to have the `registrar` column.

    Contributors:
      * @ryanbr

commit d9f93ea
Author: funilrys <[email protected]>
Date:   Sun Dec 5 17:34:48 2021 +0100

    Make regex case insensitive.

commit f7430a4
Author: Haris Gušić <[email protected]>
Date:   Fri Oct 29 13:57:02 2021 +0200

    Change delay times

    Signed-off-by: Haris Gušić <[email protected]>

commit 42286da
Author: Haris Gušić <[email protected]>
Date:   Fri Oct 29 13:56:47 2021 +0200

    Add remaining IP addresses

    Signed-off-by: Haris Gušić <[email protected]>

commit aa75d88
Author: Haris Gušić <[email protected]>
Date:   Sun Oct 17 18:39:43 2021 +0200

    Apply suggestions and create record/upload scripts

    Signed-off-by: Haris Gušić <[email protected]>

commit 8c79508
Author: Haris Gušić <[email protected]>
Date:   Sun Oct 17 01:54:11 2021 +0200

    Add automated CLI demos (proof of concept)

    Signed-off-by: Haris Gušić <[email protected]>

Author: funilrys

.github/workflows/branches.yml

@@ -114,6 +114,7 @@ jobs:
           tox -c tox.ini
 
       - name: Upload Coverage
+        continue-on-error: true
         run: |
           coveralls
         env:

.github/workflows/stable.yml

@@ -110,6 +110,7 @@ jobs:
           tox -c tox.ini
 
       - name: Upload Coverage
+        scontinue-on-error: true
         run: |
           coveralls
         env: