Cant sync mail #546

jvonau · 2021-05-13T15:54:59Z

Installed via curl install.py
fished this out of /lokole/logs/lokole_celery_worker.stderr.log

[2021-05-13 15:35:43,700: ERROR/ForkPoolWorker-1] Task opwen_email_client.webapp.tasks.sync[8af92df3-5f9b-4dda-a917-8f6d1ff96452] raised unexpected: TypeError('gethostbyname() argument 1 must be str, bytes or bytearray, not None')
Traceback (most recent call last):
File "/lokole/venv/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/lokole/venv/lib/python3.7/site-packages/celery/app/trace.py", line 704, in protected_call
return self.run(*args, **kwargs)
File "/lokole/venv/lib/python3.7/site-packages/opwen_email_client/webapp/tasks.py", line 41, in sync
if check_connection(AppConfig.STORAGE_ACCOUNT_HOST, 80):
File "/lokole/venv/lib/python3.7/site-packages/opwen_email_client/util/network.py", line 7, in check_connection
host = gethostbyname(hostname)
TypeError: gethostbyname() argument 1 must be str, bytes or bytearray, not None

pi@raspberrypi:~ $ cat /lokole/state/settings.env

OPWEN_APP_ROOT=
OPWEN_STATE_DIRECTORY=/lokole/state
OPWEN_SESSION_KEY=edit
OPWEN_MAX_UPLOAD_SIZE_MB=10
OPWEN_SIM_TYPE=Ethernet
OPWEN_EMAIL_SERVER_HOSTNAME=mailserver.lokole.ca
OPWEN_CLIENT_NAME=jv-rpi-stock
OPWEN_ROOT_DOMAIN=lokole.ca
OPWEN_RESTART_PATH=/lokole/state/lokole_restarter/lokole_gunicorn=HUP,/lokole/state/lokole_restarter/lokole_celery_worker=,/lokole/state/lokole_restarter/lokole_celery_beat=
OPWEN_SYNC_SCHEDULE=1,16,31,46
OPWEN_CLIENT_ID=edit
OPWEN_REMOTE_ACCOUNT_NAME=edit
OPWEN_REMOTE_ACCOUNT_KEY=edit
OPWEN_REMOTE_RESOURCE_CONTAINER=compressedpackages

Only the server has this variable passed to the running processes. Think this would need to be embedded in the registration reply info.

holta · 2021-05-13T21:46:28Z

@nzola is stuck likewise? In any case, he just wrote:

I cannot send or receive emails. All outgoing email from Lokole clients are stack in the outbox and all the inbound emails coming from the internet are stack in the Lokole srver, I can see the number of emaills in the server status. This situation is just with any clients I setup with ver 0.6.14. But I still have the client I created with 0.6.10. it is sill exchanging emails without any problems. I have not upgraded it yet.

jvonau · 2021-05-13T21:56:02Z

@nzola Please email privately me the contents of /lokole/state/settings.env from the 0.6.10 machine please.

nzola · 2021-05-13T22:19:52Z

@jvonau I rushed to catch the bus. I will send it late at midnight when I am home

adamsclafani · 2021-05-13T22:47:24Z

The piece of code where the error was being thrown was a new(well not so new anymore) piece that was added as part of the IIAB integration funny enough, discussion here. check_connection() basically checks for an existing connection before attempting to dial up via a USB modem.

In short, as @jvonau pointed out, the OPWEN_REMOTE_ACCOUNT_HOST variable is not set. This should also probably be returned in the register UI. That being said, @nzola is using the install script so it is definitely odd that this variable would not be set.

adamsclafani · 2021-05-13T22:59:18Z

Looking at the install script, it does not even mention or set the OPWEN_REMOTE_ACCOUNT_HOST variable. Maybe this should be 'mailserver.lokole.ca'. Maybe I should just change it it 'OPWEN_EMAIL_SERVER_HOSTNAME'?

nzola · 2021-05-14T10:45:39Z

Noticed 3 vwdial files and a directory are missing in the state directory of the new client install compared to ver 0.6.10 install

Folder for victest1 client ver 0.6.10:

pi@raspberrypi:/lokole/state $ ls
celery.cron.db lokole_celery_beat.pid settings.env wvdial.log
celery.sqlite3 lokole_gunicorn.sock users.sqlite3 wvdial.log.bak.gz
emails.sqlite3 lokole_restarter wvdial
pi@raspberrypi:/lokole/state $

Folder for victest9 client ver 0.6.14:

pi@raspberrypi:/lokole/state $ ls
celery.cron.db emails.sqlite3 lokole_gunicorn.sock settings.env
celery.sqlite3 lokole_celery_beat.pid lokole_restarter users.sqlite3
pi@raspberrypi:/lokole/state $

nzola · 2021-05-14T10:49:16Z

I don't know why those crossing lines?

holta · 2021-05-14T12:01:27Z

Thanks @nzola for having shared:

pi@raspberrypi:~ $ cd lokole
pi@raspberrypi:~/lokole $ ls
logs  state  venv
pi@raspberrypi:~/lokole $ cd state
pi@raspberrypi:~/lokole/state $ ls
celery.cron.db  emails.sqlite3          lokole_gunicorn.sock  settings.env
celery.sqlite3  lokole_celery_beat.pid  lokole_restarter      users.sqlite3
pi@raspberrypi:~/lokole/state $

@jvonau requests you post the contents of this file:

/home/pi/lokole/state/settings.env

holta · 2021-05-14T12:24:23Z

I don't know why those crossing lines?

@nzola use triple backticks ``` before and after any code block — to make it easier to read:

https://docs.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks

nzola · 2021-05-14T15:31:35Z

Folder for victest1 client ver 0.6.10:

pi@raspberrypi:/lokole/state $ ls
celery.cron.db lokole_celery_beat.pid settings.env wvdial.log
celery.sqlite3 lokole_gunicorn.sock users.sqlite3 wvdial.log.bak.gz
emails.sqlite3 lokole_restarter wvdial
pi@raspberrypi:/lokole/state $

Folder for victest9 client ver 0.6.14:

pi@raspberrypi:/lokole/state $ ls
celery.cron.db emails.sqlite3 lokole_gunicorn.sock settings.env
celery.sqlite3 lokole_celery_beat.pid lokole_restarter users.sqlite3
pi@raspberrypi:/lokole/state $

jvonau · 2021-05-14T16:02:26Z

Lets get some timestamps
ls -l /lokole/state/
ls -l /home/pi/lokole/state/
On 0.6.10 does /home/pi/lokole/state/settings.env exist? if so email me the contents as there may be sensitive info that should not be public.

jvonau · 2021-05-14T16:22:40Z

Replaced the contents of /lokole/venv/lib/python3.7/site-packages/opwen_email_client/util/network.py on jv-rpi-stock with this complete hack

def check_connection(hostname, port):
    print (hostname)
    SOMEHOST= 'mx.sendgrid.net'
    try:
        os.system("ping -c 5 " + SOMEHOST.strip(";")) is 0
        return True
    except OSError:
        pass
    except:
        return False

that would be network.py here, that is enough for check_connection to work and email starts to flow but needs polishing.

@nzola what is a valid address on vic1test that you know works?

holta · 2021-05-14T17:10:30Z

@nzola:

✅ Triple backticks: ```
❎ Triple-quotes: '''

...before and after any code block.

Make sure triple-backticks are on their own separate line:
https://docs.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks

holta · 2021-05-14T17:23:59Z

@nzola to find the backtick key (same as accent grave in French) on your keyboard, take a close look at these photos:

https://google.com/search?q=backtick&tbm=isch

(Very different from the single-quote / apostrophe!)

nzola · 2021-05-16T20:13:12Z

Lokole on IIAB install (RPI B+ 0S Buster Lite had been failing since last night at this spot.

  TASK [lokole : Disable & Stop all 4 (above) systemd services, if not lokole_enabled] ***********************************************************************************
skipping: [127.0.0.1] => (item=lokole_restarter)
skipping: [127.0.0.1] => (item=celerybeat)
skipping: [127.0.0.1] => (item=celery)
skipping: [127.0.0.1] => (item=lokole)

TASK [lokole : Enable/Disable/Restart Apache if primary] ***************************************************************************************************************
skipping: [127.0.0.1]

TASK [lokole : Enable/Disable/Restart NGINX if primary] ****************************************************************************************************************
included: /opt/iiab/iiab/roles/lokole/tasks/nginx.yml for 127.0.0.1

TASK [lokole : Enable http://box/lokole via NGINX, by installing /etc/nginx/conf.d/lokole-nginx.conf from template] ****************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'python_ver' is undefined"}

PLAY RECAP *************************************************************************************************************************************************************
127.0.0.1                  : ok=395  changed=231  unreachable=0    failed=1    skipped=85   rescued=0    ignored=4

pi@raspberrypi:~ $

jvonau · 2021-05-16T22:12:10Z

Re-open this issue, the sync issue will be uncovered once you register any new machine and is unresolved.

jvonau · 2021-05-17T09:41:21Z

worker_1  | [2021-05-16 01:38:40,245: CRITICAL/ForkPoolWorker-2] error sending email 30241279-8b4a-46cf-aa10-507fdbadcb98:{'from': {'email': '[email protected]'}, 'subject': 'simple attach', 'personalizations': [{'to': [{'email': '[email protected]'}], 'cc': [{'email': '[email protected]'}], 'bcc': [{'email': '[email protected]'}]}], 'content': [{'type': 'text/html'}], 'attachments': [{'content': 'T1BXRU5fQVBQX1JPT1Q9DQpPUFdFTl9TVEFURV9ESVJFQ1RPUlk9L2hvbWUvcGkvbG9rb2xlL3N0YXRlDQpPUFdFTl9TRVNTSU9OX0tFWT01RkdRMWk3OGU0WHlHVllKOEQ0NVBKSTFjY3dmNEJEVg0KT1BXRU5fTUFYX1VQTE9BRF9TSVpFX01CPTEwDQpPUFdFTl9TSU1fVFlQRT1FdGhlcm5ldA0KT1BXRU5fRU1BSUxfU0VSVkVSX0hPU1ROQU1FPW1haWxzZXJ2ZXIubG9rb2xlLmNhDQpPUFdFTl9DTElFTlRfTkFNRT12aWN0ZXN0OQ0KT1BXRU5fUk9PVF9ET01BSU49bG9rb2xlLmNhDQpPUFdFTl9SRVNUQVJUX1BBVEg9L2hvbWUvcGkvbG9rb2xlL3N0YXRlL2xva29sZV9yZXN0YXJ0ZXIvbG9rb2xlX2d1bmljb3JuPUhVUCwvaG9tZS9waS9sb2tvbGUvc3RhdGUvbG9rb2xlX3Jlc3RhcnRlci9sb2tvbGVfY2VsZXJ5X3dvcmtlcj0sL2hvbWUvcGkvbG9rb2xlL3N0YXRlL2xva29sZV9yZXN0YXJ0ZXIvbG9rb2xlX2NlbGVyeV9iZWF0PQ0KT1BXRU5fU1lOQ19TQ0hFRFVMRT0xLDE2LDMxLDQ2ICogKiAqICoNCk9QV0VOX0NMSUVOVF9JRD1iNTFjZTE4Zi00MjM4LTQyZjktYmM2YS02MjBkMzYyNmJiNzINCk9QV0VOX1JFTU9URV9BQ0NPVU5UX05BTUU9b3B3ZW5jbGllbnRibG9ic3YzZmYydw0KT1BXRU5fUkVNT1RFX0FDQ09VTlRfS0VZPXVBNm1vUUtpYjBNYSthRE5YY1FPM3BIN004Vjdra3doOGlWNFZUTDFVTW1OWEhGajhZUGI0ZUt5UTJMbVNLYncxTjlkWGZFYlkvWUNyWFdvOTczSmVRPT0NCk9QV0VOX1JFTU9URV9SRVNPVVJDRV9DT05UQUlORVI9Y29tcHJlc3NlZHBhY2thZ2VzDQpzZXR0aW5ncy5lbnYgKEVORCkNCg==', 'type': 'text/plain', 'filename': 'settings.env victest9.txt', 'disposition': 'attachment', 'content_id': 'settings.env victest9.txt'}], 'reply_to': {'email': '[email protected]'}}:[{'message': 'The content value must be a string at least one character in length.', 'field': 'content.0.value', 'help': 'http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#message.content.value'}] (BadRequestsError(<HTTPError 400: 'Bad Request'>))

Think the above is because there is a space in the attachment filename 'content_id': 'settings.env victest9.txt' @nzola Did the email and attachment go though to all recipients?

worker_1  | [2021-05-17 02:16:08,948: INFO/ForkPoolWorker-2] sent email fc9e0956-088d-46dc-9b22-b6b54b6ebdb3
worker_1  | [2021-05-17 02:16:08,948: INFO/ForkPoolWorker-2] email_delivered_from_client|{'domain': 'jv-rpi-stock.lokole.ca'}
worker_1  | [2021-05-17 02:16:08,949: INFO/ForkPoolWorker-2] Task opwen_email_server.integration.celery.send[fbf3e91a-8076-493a-894e-d511947fa425] succeeded in 0.23927182145416737s: None
worker_1  | [2021-05-17 02:31:03,555: INFO/MainProcess] Received task: opwen_email_server.integration.celery.written_store[5712513e-9350-48d8-88c2-f34a0df317dc]  
worker_1  | [2021-05-17 02:31:03,968: INFO/ForkPoolWorker-4] email_stored_from_client|{'domain': 'jv-rpi-stock.lokole.ca', 'num_emails': 1}
worker_1  | [2021-05-17 02:31:04,052: INFO/ForkPoolWorker-4] user_stored_from_client|{'domain': 'jv-rpi-stock.lokole.ca', 'num_users': 2}
worker_1  | [2021-05-17 02:31:04,084: INFO/ForkPoolWorker-4] Task opwen_email_server.integration.celery.written_store[5712513e-9350-48d8-88c2-f34a0df317dc] succeeded in 0.52700000628829s: None
worker_1  | [2021-05-17 02:31:08,570: INFO/MainProcess] Received task: opwen_email_server.integration.celery.send[92d2dcfe-a01a-42a5-b865-65e55118bd9c]  
worker_1  | [2021-05-17 02:31:08,711: INFO/ForkPoolWorker-3] sent email bdb32cb6-d75c-4d68-b5f6-ce99fd61a7f7
worker_1  | [2021-05-17 02:31:08,711: INFO/ForkPoolWorker-3] email_delivered_from_client|{'domain': 'jv-rpi-stock.lokole.ca'}
worker_1  | [2021-05-17 02:31:08,711: INFO/ForkPoolWorker-3] Task opwen_email_server.integration.celery.send[92d2dcfe-a01a-42a5-b865-65e55118bd9c] succeeded in 0.13928200863301754s: None
worker_1  | [2021-05-17 02:31:09,257: INFO/MainProcess] Received task: opwen_email_server.integration.celery.index_received_email_for_mailbox[499c9807-8dd3-4d7a-ad26-694d81a3330c]  
worker_1  | [2021-05-17 02:31:09,282: INFO/MainProcess] Received task: opwen_email_server.integration.celery.index_sent_email_for_mailbox[b9779204-05d0-44cb-8e41-c7109c57ad9c]  
worker_1  | [2021-05-17 02:31:09,300: CRITICAL/ForkPoolWorker-5] IndexReceivedEmailForMailbox(args=('bdb32cb6-d75c-4d68-b5f6-ce99fd61a7f7',), kwargs={}) (KeyError('sent_at'))
worker_1  | [2021-05-17 02:31:09,301: ERROR/ForkPoolWorker-5] Task opwen_email_server.integration.celery.index_received_email_for_mailbox[499c9807-8dd3-4d7a-ad26-694d81a3330c] raised unexpected: KeyError('sent_at')
worker_1  | Traceback (most recent call last):
worker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
worker_1  |     R = retval = fun(*args, **kwargs)
worker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
worker_1  |     return self.run(*args, **kwargs)
worker_1  |   File "/app/opwen_email_server/integration/celery.py", line 55, in index_received_email_for_mailbox
worker_1  |     action(resource_id)
worker_1  |   File "/app/opwen_email_server/actions.py", line 43, in __call__
worker_1  |     raise ex
worker_1  |   File "/app/opwen_email_server/utils/log.py", line 57, in log_exception
worker_1  |     raise ex
worker_1  |   File "/app/opwen_email_server/actions.py", line 40, in __call__
worker_1  |     return self._action(*args, **kwargs)
worker_1  |   File "/app/opwen_email_server/actions.py", line 126, in _action
worker_1  |     desc_prefix = descending_timestamp(email['sent_at'])
worker_1  | KeyError: 'sent_at'
worker_1  | [2021-05-17 02:31:09,324: CRITICAL/ForkPoolWorker-1] IndexSentEmailForMailbox(args=('bdb32cb6-d75c-4d68-b5f6-ce99fd61a7f7',), kwargs={}) (KeyError('sent_at'))
worker_1  | [2021-05-17 02:31:09,325: ERROR/ForkPoolWorker-1] Task opwen_email_server.integration.celery.index_sent_email_for_mailbox[b9779204-05d0-44cb-8e41-c7109c57ad9c] raised unexpected: KeyError('sent_at')
worker_1  | Traceback (most recent call last):
worker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
worker_1  |     R = retval = fun(*args, **kwargs)
worker_1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
worker_1  |     return self.run(*args, **kwargs)
worker_1  |   File "/app/opwen_email_server/integration/celery.py", line 65, in index_sent_email_for_mailbox
worker_1  |     action(resource_id)
worker_1  |   File "/app/opwen_email_server/actions.py", line 43, in __call__
worker_1  |     raise ex
worker_1  |   File "/app/opwen_email_server/utils/log.py", line 57, in log_exception
worker_1  |     raise ex
worker_1  |   File "/app/opwen_email_server/actions.py", line 40, in __call__
worker_1  |     return self._action(*args, **kwargs)
worker_1  |   File "/app/opwen_email_server/actions.py", line 126, in _action
worker_1  |     desc_prefix = descending_timestamp(email['sent_at'])
worker_1  | KeyError: 'sent_at'
worker_1  | [2021-05-17 05:50:12,200: INFO/MainProcess] Received task: opwen_email_server.integration.celery.inbound_store[ba75919e-795c-4ed3-81c1-894ea7691a44]  
worker_1  | [2021-05-17 05:50:12,520: INFO/ForkPoolWorker-2] email_stored_for_client|{'domain': 'gmail.com'}
worker_1  | [2021-05-17 05:50:12,520: INFO/ForkPoolWorker-2] Task opwen_email_server.integration.celery.inbound_store[ba75919e-795c-4ed3-81c1-894ea7691a44] succeeded in 0.318214894272387s: None

These are 2 emails are outside world <-> remote lokole, the emails are sent and received on the the remote's end to the outside world and back, this looks like a copy of the email are kept somewhere else, for the web-portal maybe?

jvonau · 2021-05-17T11:06:15Z

Remote site

Web portal

adamsclafani · 2021-05-22T23:24:16Z

Looks like the sent_at field is missing when the email was created. I'll look into this one as well. Should be a relatively easy fix. Thanks so much, these logs help big time

nzola · 2021-05-22T23:29:38Z

Yes. That is correct. Any email received in gmail is missing @

…

On Sat, May 22, 2021 at 4:24 PM Adam Sclafani ***@***.***> wrote: Looks like the sent_at field is missing when the email was created. I'll look into this one as well. Should be a relatively easy fix. Thanks so much, these logs help big time — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#546 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEH5MMYBZMBEPFA3JWRIF3TTPA4KZANCNFSM442ZO3VA> .

adamsclafani · 2021-05-28T20:48:26Z

@jvonau did you send an email from an outside provider to a lokole email or the other way around to trigger this?

After doing a code trace, all emails created with lokole should have the 'sent_at' field upon creation.

jvonau · 2021-05-28T22:27:50Z

@jvonau did you send an email from an outside provider to a lokole email or the other way around to trigger this?

The top error was victest's test client. the second one was definitionally me outside <> client, but lets just keep an eye on the server, could be resolved over the last 2 weeks with the client side refinements.

After doing a code trace, all emails created with lokole should have the 'sent_at' field upon creation.

Agreed, so I'm not sure if this is a client side issue when the email was created or on the server when a email is stored before forwarding, just have to eyeball the all logs for a awhile.

This was referenced May 13, 2021

Lokole 0.6.11 & 0.6.12 released, but pip install not yet available [0.6.13, 0.6.14, 0.6.15, 0.6.16 & 0.8.0 have restored the pip install option. Need to finalize OPWEN variables' settings] iiab/iiab#2767

Closed

Implement Terraform #512

Open

nzola closed this as completed May 16, 2021

holta mentioned this issue May 16, 2021

Lokole install blocked on RaspiOS, so add 'python_ver: 3.7' to vars/raspbian-10.yml iiab/iiab#2780

Merged

adamsclafani mentioned this issue May 16, 2021

Bug Fixes #547

Merged

nzola reopened this May 16, 2021

holta mentioned this issue May 19, 2021

Further clarify lokole_sim_type options iiab/iiab#2788

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cant sync mail #546

Cant sync mail #546

jvonau commented May 13, 2021 •

edited

Loading

holta commented May 13, 2021

jvonau commented May 13, 2021

nzola commented May 13, 2021

adamsclafani commented May 13, 2021 •

edited

Loading

adamsclafani commented May 13, 2021

nzola commented May 14, 2021

nzola commented May 14, 2021

holta commented May 14, 2021 •

edited

Loading

holta commented May 14, 2021

nzola commented May 14, 2021 •

edited

Loading

jvonau commented May 14, 2021

jvonau commented May 14, 2021

holta commented May 14, 2021

holta commented May 14, 2021

nzola commented May 16, 2021 •

edited

Loading

jvonau commented May 16, 2021

jvonau commented May 17, 2021 •

edited

Loading

jvonau commented May 17, 2021

adamsclafani commented May 22, 2021

nzola commented May 22, 2021 via email

adamsclafani commented May 28, 2021

jvonau commented May 28, 2021

Cant sync mail #546

Cant sync mail #546

Comments

jvonau commented May 13, 2021 • edited Loading

holta commented May 13, 2021

jvonau commented May 13, 2021

nzola commented May 13, 2021

adamsclafani commented May 13, 2021 • edited Loading

adamsclafani commented May 13, 2021

nzola commented May 14, 2021

nzola commented May 14, 2021

holta commented May 14, 2021 • edited Loading

holta commented May 14, 2021

nzola commented May 14, 2021 • edited Loading

jvonau commented May 14, 2021

jvonau commented May 14, 2021

holta commented May 14, 2021

holta commented May 14, 2021

nzola commented May 16, 2021 • edited Loading

jvonau commented May 16, 2021

jvonau commented May 17, 2021 • edited Loading

jvonau commented May 17, 2021

adamsclafani commented May 22, 2021

nzola commented May 22, 2021 via email

adamsclafani commented May 28, 2021

jvonau commented May 28, 2021

jvonau commented May 13, 2021 •

edited

Loading

adamsclafani commented May 13, 2021 •

edited

Loading

holta commented May 14, 2021 •

edited

Loading

nzola commented May 14, 2021 •

edited

Loading

nzola commented May 16, 2021 •

edited

Loading

jvonau commented May 17, 2021 •

edited

Loading