Improve URL boundary regarding quotation marks and parentheses

[fhightower]

The following tests were failing and I'd like to get them passing:

s = "DownloadString('https://example[.]com/rdp.ps1');g $I"
iocs = find_iocs(s)
assert iocs['urls'] == ['https://example.com/rdp.ps1']

s = 'DownloadString("https://example[.]com/rdp.ps1");g $I'
iocs = find_iocs(s)
assert iocs['urls'] == ['https://example.com/rdp.ps1']

[fhightower]

Another example:

find_iocs('Foo https://citizenlab.ca/about/), bar')

finds https://citizenlab.ca/about/), as a URL when https://citizenlab.ca/about/ is expected

[FANGOD]

I also have a problem.

A line of file:
url,https://groups.google.com/g/vfc9gs,Malicious Google Groups discussion
but got:
urls=["https://groups.google.com/g/vfc9gs,Malicious"]

[fhightower]

Thanks for reporting this, @FANGOD. I hope to have this fixed soon.

[fhightower]

This issue is really tracking two things and I want to split them up as one is relatively easy to solve and the second requires some thought.

I've renamed this issue to focus on problems with the URL boundary regarding single quotation marks and parenthesis. For example, given Foo 'https://citizenlab.ca/about/'), bar, we expect to parse https://citizenlab.ca/about/ as the URL (not the originally parsed https://citizenlab.ca/about/')).

The other challenge with URL boundaries in the issue you reported @FANGOD about commas. This is tricky as noted here because commas are technically a valid part of a URL path. I'm going to track updates to the way commas are handled in #261 .