Bulk add spam rules in a single call

[Bilal-io]

I am working on a browser extension to help me bulk manage spam emails. I plan to open source it and share it with the community.
I need a way to bulk add spam rules.

This should not break any existing API, the purpose of this is to add a new method in the CustomerFacade that I can interact with through the extension's contentscript.

I could just call entityClient.update directly, but I did not want to handle generating the hashedValue myself.

PS: I'd be happy to polish my implementation of the bulk spam management and create a PR to integrate it into the product if I get some feedback from the community and Tutanota's devs. I'll post in Reddit once ready.

[Bilal-io]

Please change the tag to small improvement or improvement.

[charlag]

Hi. I will convert this to discuss because there are few points we need to discuss

[charlag]

So, may I ask what is the extension and what are you trying to achieve with it? Maybe it's something worth implementing directly?
I also have concerns about adding too many spam rules.

[Bilal-io]

Hey, thank you converting this to a discussion, because that can help me achieve my results better.

My main issue is that I get a lot of phishing and spam emails, i noticed that the more I am actively adding spam rules the less I get, but once I ignore it for a couple of weeks, I start getting more and more of them.

The reason I ignore them sometimes is because of how time consuming it is to add a single spam rule, and therefore I wanted to automate the process — but not fully, since I wouldn't want to add a legitimate domain such as gmail.com or amazonses.com to the spam rules.

From your concern, I understand that many spam filter can slow things down? I'd like to know what's an alternative. If you guys provided me with an API, or a way to script, I could try to deal with the spam in a better way.

So my extension would interact with the UI, I would add a button next to the Clear All button in the Spam folder. Let's say "Bulk Manage Spam", once hit, I'd display the list of emails in full width, extract and display the emails and domains on the right side with check boxes next to each to report, add spam rule and delete, once I double check there are no legitimate domains that I would mistakenly add to spam rules, I can go ahead and click a single button that does all of this for me for each from email, alt email and each domain and subdomain of the spammer.

Happy to clarify anything, and answer any questions.

[charlag]

Hey, thanks a lot for the reply.

First of all, we probably need to improve our spam rules in this situation and we are already working towards that. When you report emails as spam or not spam it already helps a lot. Spam rules are mostly meant to override the spam filter and not to fix it. I think if you would pitch this improvement to the team they will answer the same. The thing is, the spam rules are applied on the server by comparing hashes and it's not really meant to do too much work.

Are you a Premium customer? It might be worth talking to them about the cases where you receive spam and if we can find more general solution.

If you still want to implement your workaround you can use built-in WebCrypto to do SHA256 easily.

[Bilal-io]

Yes, I am a premium customer. Who am I supposed to reach out to? Can the relevant individuals join the discussion here? I'd be happy to share my use cases with them.

The problem with Tutanota's WebCrypto is that it's a library, bundled and inaccessibly from the browser (I simply access the APIs from the global tutao variable in the browser). Also, I am trying not to have any dependencies in my implementation so that people can trust it more. I tried accessing the methods in https://mail.tutanota.com/common-52449c87.js using System global variable, but the methods in it are all obfuscated.

I will share a draft of the script I wrote and will be using for the extension. Hopefully it helps explain things better.

[charlag]

There's a Support button in the left narrow column. It's best to discuss individual cases and share examples there, you can also link this discussion.
I didn't mean our sha256 impl, I meant WebCrypto from browser

[Bilal-io]

I'll try to contact them soon.
And thanks for the link, I'll evaluate the WebCrypto.

[Bilal-io]

The first part

const SpamRuleFieldType = {
	FROM: "0",
	TO: "1",
	CC: "2",
	BCC: "3"
}
const MailReportType = {
	PHISHING: "0",
	SPAM: "1"
}
const SpamRuleType = {
	WHITELIST: "1",
	BLACKLIST: "2",
	DISCARD: "3"
}
// must vet list of emails in the spam folder first
// move any false negatives over to the inbox
await(async () => {
	const currentList = tutao.currentView.mailList.mailView.mailList.list;

	const showingTrashOrSpamFolder = () => {
		return tutao.currentView.mailList.mailView.selectedFolder.folderType === "5";
	}

	const recursivelyLoadSpamEntities = async () => {
		await currentList._loadMore()
		currentList._domList.style.height = currentList._calculateListHeight();

		if (currentList._loadedCompletely) {
			console.log("Done loading all spam!");
			return;
		} else {
			recursivelyLoadSpamEntities();
		}
	}

	const getSubdomains = (subdomain, extractedSubdomains = []) => {
		let domainSegmentsArray = [];
		do {
			extractedSubdomains.push(subdomain);
			domainSegmentsArray = subdomain.split('.');
			domainSegmentsArray.shift();
			subdomain = domainSegmentsArray.join('.');
		} while (domainSegmentsArray.length >= 2)
		return extractedSubdomains;
	}

	if (showingTrashOrSpamFolder()) {
		// use a set to store emails and domains to report and add to spam filter, this ensures we have unique values.
		let emailsNDomainsList = new Set();
		// load all spam emails
		await recursivelyLoadSpamEntities();
		// get loaded spam emails
		const loadedSpamEntities = currentList.getLoadedEntities();

		// report phishing to tutanota
		tutao.locator.mailModel.reportMails(MailReportType.PHISHING, loadedSpamEntities);

		// parse emails to domains
		loadedSpamEntities.forEach(entity => {
			// add From email
			emailsNDomainsList.add(entity.sender.address);
			// add From domain
			emailsNDomainsList.add(entity.sender.address.split('@')[1]);
			// check for an alt sender email
			const differentEnvelopeSenderEmail = entity.differentEnvelopeSender;
			if (differentEnvelopeSenderEmail) {
				// add alt email to the set
				emailsNDomainsList.add(differentEnvelopeSenderEmail);
				// check if the alt domain is a subdomain, usually mail.shittydomain.poop
				const segments = differentEnvelopeSenderEmail.split('@')[1].split('.');
				for (let i = 0; i < segments.length - 1; i++) {
					emailsNDomainsList.add(segments.slice(i).join('.'))
				}
			}
		})

		// get existing spam rules
		tutao.locator.customerFacade.loadCustomerServerProperties().then(csp => {
			csp.emailSenderList.forEach(r => {
				if (r.type === SpamRuleType.BLACKLIST || r.type === SpamRuleType.DISCARD) {
					// filter out emails and domains already added to spam rule
					// could use .has to check first, but using delete is safe
					if (emailsNDomainsList.has(r.value)) console.log(r.value)
					emailsNDomainsList.delete(r.value);
				}
			})
			console.log([...emailsNDomainsList]);
		})
	}
})();

The second part

// Must vet the list generated above in case of false negatives. Wo you don't block domains such as gmail, yahoo, amazonses or any other trusted domains that spammers abuse.
let manuallyVettedList = [];
await(async () => {
	const customerFacade = tutao.locator.customerFacade;

	// loops through list of manually vetted emails and domains
	const spamRulesList = [];
	manuallyVettedList.forEach(emailOrDomain => {
		// add spam rule to list
		spamRulesList.push({value: emailOrDomain, field: SpamRuleFieldType.FROM, type: SpamRuleType.DISCARD});
	});

	// this is where my PR helps.
	customerFacade.addSpamRules(spamRulesList)
}) ();

[Bilal-io]

PR closed, since I am able to create identical hashed values using the Web Crypto API. Thanks @charlag for the tip

const _buffer = new TextEncoder("utf-8").encode("testdomainnameofspammer.com");
const _uint8Array = crypto.subtle.digest("SHA-256", buffer).then(s =>  s);

const base64_arraybuffer = async (data) => {
    const _base64url = await new Promise((r) => {
        const _reader = new FileReader()
        _reader.onload = () => r(_reader.result)
        _reader.readAsDataURL(new Blob([data]))
    });

    return _base64url.split(",", 2)[1];
}

const hashedValue = await base64_arraybuffer(_uint8Array);

[charlag]

Glad it worked. I think there are some problems with your snippet though. You are passing promise to base64_arraybuffer and I'm not sure why you need FileReader...

I hope in the long term this can be solved without mass spam rules