Conflicting with our firewall policies - Some users cannot access Google Workspace mail

[blowrancebenton]

This appears to causing an issue for some of our on-prem clients behind our Sonicwall firewall. We are a school district and block external email access along with proxy servers and other site categories, applications, etc. Some of our users are getting error messages attempting to access their Google Workspace email. After much troubleshooting it seems we've isolated it to Google ChromeVariations being enabled and/or the "IP Protection Proxy" flag being Default (We do not know the Active Variation GUID to look for that on systems that our users report the issue).

Example of user complaints:

• Issues were reported late this afternoon from {redacted} and {redacted} regarding an issue accessing email. {redacted} stated the issue started the week before Thanksgiving and has persisted.. Error message: This site can't be reached ... "The webpage at https://mail.google.com/mail/u/0/?authuser=0 might be temporarily down or it may have moved permanently to a new web address." "ERR_FAILED"

• Email can not pull up. Keeps saying there is a firewall or proxy that is keeping it from working. Have reset my connection, cleared cache, and restarted computer.

Resolution for us is to:

1. Disable Google ChromeVariations (or set to enable for critical fixes only) and/or
2. Go into chrome://flags and disable "IP Protection Proxy"

Once we do this, we can terminate and relaunch Chrome and then the user can access email again.

Our first discovery of this was November 30, 2023 which reportedly started around November 20-24. This seems to be a growing issue with the above solution resolving each one.

[miketaylr]

Hey @blowrancebenton - really appreciate you opening this issue.

Would you be able to reach out to some of the affected users and ask what versions of Chrome they're running? If you would prefer, feel free to send an reply email to [email protected] - or here in the github issue is fine as well.

[blowrancebenton]

One user in our tech department is affected. He can produce and resolve the issue by reversing the resolution mentioned or applying it. He is on 120.0.6099.225 on Windows 10 22H2.

[DavidSchinazi]

Thanks for the details @blowrancebenton. Could you ask the user to please collect a network log in both the working and broken states please? That would be incredibly helpful for us to debug. Please send the log to Mike (see email above) or open a new issue on crbug.com, whichever is simplest for you

[blowrancebenton]

Requested logs emailed. One log is with ChromeVariations and IP Protection Proxy enabled where Google Workspace Mail is not loading and the other is with these disabled and Google Workspace Mail working.

[DavidSchinazi]

Thank you! We'll look at these internally and report back.

[recvfrom]

Hi @blowrancebenton, I tried to follow up with you via email but haven't heard back yet, so I figured I'd post here as well.

I looked at the network logs provided and it seems that when variations / IP Protection are enabled, the network log shows that Chrome attempts to access a mail.google.com URL but the connection is refused before Chrome can successfully establish an SSL connection to it. The other network log only shows that connections to mail.google.com re-use an existing connection, so it's unclear why an SSL connection attempt succeeded in that case. It doesn't seem that proxy settings or DNS information is different in both network logs, and it doesn't seem that PAC scripts are used on your network to configure a proxy.

Can you provide additional information on how external email access is blocked on your network? Does this include access to personal mail.google.com accounts? If so I'm wondering how that is implemented (specifically, if the firewall is resetting connections when it sees attempts to connect to mail.google.com in one network log, I'm wondering why it wouldn't in the second network log). Do you use enterprise policy or a chrome extension to configure a proxy for certain traffic through your firewall?

Feel free to post back here or reach out via email. Thanks!

[blowrancebenton]

In further testing, we found that "Disable IP Protection Proxy" on an affected computer could be enabled or disabled and the issue remained so now I'm back tracking and thinking this issue may not be directly related to this project. When this system was unable to access mail.google.com, we found that despite ChromeVariations being disabled or set to critical fixes only the Chrome://version page unexpectedly showed a long list of variations again. Disabling/Enabling ChromeVariations by toggling the registry key with values of either 1 or 2, stopping and relaunching Chrome had a direct impact on being able to access mail.google.com on that system.

…

On Wed, Feb 7, 2024 at 3:04 PM Andrew Williams ***@***.***> wrote: Hi @blowrancebenton <https://github.com/blowrancebenton>, I tried to follow up with you via email but haven't heard back yet, so I figured I'd post here as well. I looked at the network logs provided and it seems that when variations / IP Protection are enabled, the network log shows that Chrome attempts to access a mail.google.com URL but the connection is refused before Chrome can successfully establish an SSL connection to it. The other network log only shows that connections to mail.google.com re-use an existing connection, so it's unclear why an SSL connection attempt succeeded in that case. It doesn't seem that proxy settings or DNS information is different in both network logs, and it doesn't seem that PAC scripts are used on your network to configure a proxy. Can you provide additional information on how external email access is blocked on your network? Does this include access to personal mail.google.com accounts? If so I'm wondering how that is implemented (specifically, if the firewall is resetting connections when it sees attempts to connect to mail.google.com in one network log, I'm wondering why it wouldn't in the second network log). Do you use enterprise policy or a chrome extension to configure a proxy for certain traffic through your firewall? Feel free to post back here or reach out via email. Thanks! — Reply to this email directly, view it on GitHub <#40 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFX4PHO54N52RWQZIQJRKLYSPT7PAVCNFSM6AAAAABCHQUJKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSHA4TOMJVGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Disclaimer: The information in this email may be privileged, confidential and protected from disclosure. If the reader of this email is not the intended recipient, you are hereby notified that any dissemination, distribution, copying or other use of this email is strictly prohibited. If you have received this email in error, please notify the sender immediately by email and delete the material from all devices. Benton School District, 207 W. Conway, Benton, AR 72015. www.bentonschools.org <http://www.bentonschools.org>. 501-778-4861.