Miscellaneous Arlo warnings and errors #13176

arsaboo · 2018-03-13T13:26:24Z

Home Assistant release (hass --version): 0.65.3

Python release (python3 --version): 3.6.2

Component/platform: Arlo

Description of problem: Looks like our Arlo platform needs some TLC. Currently, the the Arlo integration slows down the system and generates lots of errors/warnings as below:

2017-11-15 08:35:03 WARNING (MainThread) [homeassistant.setup] Setup of alarm_control_panel is taking over 10 seconds.
2017-11-15 08:35:03 WARNING (MainThread) [homeassistant.setup] Setup of camera is taking over 10 seconds.
2017-11-15 08:35:05 WARNING (MainThread) [homeassistant.components.alarm_control_panel] Setup of platform arlo is taking over 10 seconds.
2017-11-15 08:35:05 WARNING (MainThread) [homeassistant.components.camera] Setup of platform arlo is taking over 10 seconds.

These are not new issues and have been reported several times on the forums
https://community.home-assistant.io/t/insteon-hub-pro-help/31276/8
https://community.home-assistant.io/t/hassio-io-10-second-warnings/23714/11

I did some digging and added So, I added _LOGGER.debug("Arlo component update") after this statement and _LOGGER.debug("Arlo sensor update") after this one and here is the log:

2018-03-12 20:34:36 DEBUG (Thread-7) [custom_components.arlo] Arlo component update
2018-03-12 20:34:37 DEBUG (Thread-2) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-9) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-3) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-10) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-11) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-4) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-6) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-12) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-13) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-5) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-14) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:34:37 DEBUG (Thread-8) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-19) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-20) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-21) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-13) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-18) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-4) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-8) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-2) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-7) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-9) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-12) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:36:11 DEBUG (Thread-14) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-11) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-3) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-15) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-16) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-17) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-5) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-13) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-18) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-6) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-7) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-12) [custom_components.sensor.arlo] Arlo sensor update
2018-03-12 20:37:42 DEBUG (Thread-21) [custom_components.sensor.arlo] Arlo sensor update

Looks like each sensor has its own log entry, suggesting that every sensor has its own Arlo object that it updates every time. We need to change that so that all sensors use one Arlo data object, which is updated every 30 seconds or so.

cc: @tchellomello

The text was updated successfully, but these errors were encountered:

tchellomello · 2018-03-13T13:45:25Z

Adding myself to monitor this issue

jwillaz · 2018-03-13T16:45:23Z

I think I figured out a solution. I tested these changes directly. Don't have time at the moment to set up Git and do PR right now, but if someone wants to run with these changes.

from homeassistant.util import Throttle

Fix for battery drain:
SCAN_INTERVAL = timedelta(seconds=3600)

Add above "def update(self)" in camera/arlo.py and sensor/arlo.py:
@Throttle(SCAN_INTERVAL)

ALSO, @tchellomello, the logic I added in base_station.py in the Pyarlo library for doing makeshift throttling should be removed:

current_time = int(time.time())
last_refresh = 0 if self._last_refresh is None else self._last_refresh

if current_time >= (last_refresh + self._refresh_rate):

Since we'll be using @Throttle, _refresh_rate and _last_refresh should not be needed. Apologies for my poor understanding of how throttling works in home assistant. If there is a better way to handle this, by all means go for it.

My only concern is making sure each call to an individual camera via either the Camera component or Sensor component isn't cascading via the Base Station into multiple calls to ALL cameras. I think @Throttle on update should help with that. SCAN_INTERVAL being set to 1 hour (especially for the Sensor component) will help with battery drain because having it set to 90 seconds means that the base station is waking up each camera every 90 seconds for info updates. Camera settings and values (like signal strength, battery status, etc) shouldn't be updating that often.

jwillaz · 2018-03-13T17:58:55Z

There may be something additional at play here. After a while, I started receiving the "____ is taking over 10 seconds" warnings from both sensor and camera components.

arsaboo · 2018-03-17T13:13:05Z

It seems Arlo has completely stopped working off late. Also, see https://community.home-assistant.io/t/arlo-cameras-and-sensors-entities-not-showing-up/33684/16

JBenzie · 2018-03-19T12:51:30Z

I've also noticed an issue with my Arlo camera. I've been receiving the various warnings in the logs as stated above, however, today it stopped working altogether. There are now log entries stating each component is failing.

mwd102 · 2018-03-25T02:10:26Z

New to Hass and an Arlo owner, having the same issues. I think I have a fix for the connection pool discards, PR here: #13445

Short of it is pyarlo is not designed with asynchronicity in mind - increased thread pool and set threads to block so that requests coming in when a pool is full (happens often, thanks to netgears slow response times) it waits instead of discarding.

mwd102 · 2018-03-25T03:13:50Z

Taking a look into the updating -

Each sensor has a reference to the same PyArlo and thus the same ArloBaseStation (so far so good).

The issue is with the async code, the update method will be called from however many sensors the user has. The check for the last_refresh will eval to true partly because the time is rounded to an int and partly because of the time it takes to make the web request and set the last update before other threads get there.

A hacky fix to prove the concept was to set a lock in the update function in base_station.py of the pyarlo module:

def update(self):
        if self._update_lock:
            while self._update_lock:
                time.sleep(0.5)
                if not self._update_lock:
                    return self.camera_properties

        self._update_lock = True
        current_time = time.time()

        last_refresh = 0 if self._last_refresh is None else self._last_refresh

        if current_time >= (last_refresh + self._refresh_rate):
            self.get_cameras_properties()
            self._attrs = self._session.refresh_attributes(self.name)
            _LOGGER.debug("Called base station update of camera properties: "
                          "Scan Interval: %s, New Properties: %s",
                          self._refresh_rate, self.camera_properties)
        self._update_lock = False

this means only the first sensor to ask for the update will succeed in a web request. in order to solve the issue of returning up to date info to all the others, i just parked them with a sleep and return when the thread that is making the request updates the data and releases the lock. like i said, PoC - interested in others thoughts?

jwillaz · 2018-03-27T03:35:51Z

@mwd102 - you are correct about there being issues in the update methods. I originally added that timing logic as a rudimentary way of reducing the number of calls made to the Arlo endpoints. It worked in that regard, and obviously sometimes you would still get multiple calls because there was no real locking mechanism. I have, however, been using the following modified code to tighten that up a bit more:

def update(self):
        """Update object properties."""
        current_time = int(time.time())
        last_refresh = 0 if self._last_refresh is None else self._last_refresh

        with self._update_lock:
            if current_time >= (last_refresh + self._refresh_rate):
                self.get_cameras_properties()
                _LOGGER.debug("Called base station update of camera properties: "
                              "Scan Interval: %s, New Properties: %s",
                              self._refresh_rate, self.camera_properties)
                self._session.refresh_attributes()
                self._last_refresh = int(time.time())
                for device in self._session._device_data:
                    if device.get('deviceName') == self.name:
                        self._attrs = device
                        return
        return

Even with these modifications, I still receive the warnings/errors in question. Your approach may be a better one. I am very novice with Python, but I love HASS and really appreciate the work folks like @tchellomello have done to get the Arlo component started, especially with no official API support from Netgear. I hope more people, especially those with stronger Python skills than mine, will get involved.

From my perspective, the biggest thing causing issues right now is the difference in concept of a "camera" entity in HASS vs what that means in Arlo systems. Typically, a Camera would be some singularly addressable entity with its own endpoint, attributes, stream, JPEG captures, etc. This is sort of the case with standalone Arlo cameras like Arlo Q, Arlo Baby, and Arlo Go, but not so much with Arlo and Arlo Pro which are dependent on the base station. Each of the "camera" entities for Arlo and Arlo Pro is really just a mock-up of data being retrieved from the Base Station. A single call for update to the Base Station will contain most of the new information for ALL cameras, which is why I wanted to throttle these update calls in the first place: pending you have more than one camera, more than one call to the base station is unnecessary. And then there are "other" camera attribute update calls on top of that. Confusing.

mwd102 · 2018-03-27T17:34:55Z

@jwillaz You're totally right - a the concept of camera in pyarlo (for Pro at least) and sensor in HASS are not equivalent which is causing issues. I need to look into the HASS state machine and signalling to see if we can come up with something better. Have the base station call update() and cascade to the sensor 'camera'.

The other errors you are getting might be fixed by my other suggestion in the PR which is below. HASS rejected that as they feel (rightly) that this is probably better placed upstream. Try adding the below onto line 46 of components/arlo.py

(from requests.adapters import HTTPAdapter)

arlo.session.mount('https://',HTTPAdapter(pool_maxsize=10,pool_block=True))

I don't see any indication that I am making more than one request with those two patches, neither do I get too slow warns or connection discards. Totally accept this is probably not a finished solution. Would be really interested in @tchellomello thoughts on wether we should update pyarlo or work something into HASS.

jurgenweber · 2018-04-10T03:56:47Z

I have an Arlo Baby and nothing works.. Someitmes when I restart the camera might show one photo but all of the sensors are blank. Would this be related or another problem and I should make another issue?

the interesting part is I have no logs... even after adding this to my logger config:

logger:
  default: warn
  logs:
    homeassistant.components.automation: info
    homeassistant.components.arlo: info
    homeassistant.components.camera.arlo: info
    homeassistant.components.sensor.arlo: info

codypet · 2018-04-17T15:30:35Z

@jurgenweber I'm having the exact same problem. I believe its related because I'm getting the warnings under info. I was starting to have these problems when I went to 0.65.6, but the kicker is that it started working fine randomly until I updated to release 0.67. I hadn't done anything to my Hass in 2 weeks and then one day it just started working. I'm wondering if the same thing will happen again in a week.

jurgenweber · 2018-04-19T06:51:21Z

yeah, I still get nothing. Shame no one else in the thread has piped up. I will debug more in the near future.

jurgenweber · 2018-04-21T11:46:29Z

ok, after some investigation...... I found out:

tchellomello/python-arlo#73

but I feel that this project is now a littler dead..... I then found this one; https://github.com/jeffreydwalter/arlo which seems to be much more active.

I am thinking we should swap to the more active project for the integrations and move on, etc. If I have some time I will look into that deeper.

jeffreydwalter · 2018-04-21T23:00:31Z

@jurgenweber I do actively maintain my library and welcome any contributions.

I am happy to do what I can to add support for any missing features in my library. I only own the first generation Arlo cameras, so any features not directly related to those cameras will require assistance from others to add and test. That being said, my library does work for Arlo, Arlo 2, Arlo Pro, Arlo Q, and Arlo GO. I'm sure there are probably some minor features that are not currently exposed via the API in my library, but when they are brought to my attention I do my best to add them.

arsaboo · 2018-05-01T19:24:17Z

@jurgenweber @jeffreydwalter Can this new library be used in a custom component right now?

tchellomello · 2018-05-01T21:23:50Z

Hey guys, it's been a while since I last touch on the pyarlo lib. @jeffreydwalter how about if we merge both projects and rebase the HA component using only one project?

rafuz · 2018-05-20T22:47:45Z

Hello guys, any news on this topic?

jurgenweber · 2018-05-21T01:42:02Z

I started and I have on my machine some sensors using @jeffreydwalter library but my boy was born and I have no time. The sensors were taking too long to update giving a warning and I wanted to look into any alternative approach for updating them instead of querying arlo.netgear.com every time but by actually listening to the stream from the arlo and updating the sensors on the fly..... I am finding it hard to work out the HaSS eco system as this is the first time I have tired to do something in it as well. .... :\ I would really love something thou.

tchellomello · 2018-05-21T04:56:46Z

Guys, I'm going to work on this issue this week and see if we can do anything to address that.

tchellomello · 2018-05-29T07:08:55Z

Guys, I've started working today on a new PR (still local only) that fixed this problem. Actually, we are using now some cache states which will reduce a lot of queries to the Arlo API.

Just give me a few more days and I'll submit a PR with some test instructions so then can you test it too.

jurgenweber · 2018-05-29T07:49:29Z

yay, looking forward to it and I am happy to test.

grantclem · 2018-05-29T10:08:49Z

also happy to test, thanks for your effort resolving this

jwillaz · 2018-05-31T18:48:40Z

@tchellomello Do you know how to make the SCAN_INTERVAL dynamic (set in yaml config) for the camera & sensor components? Right now they are set to 90 seconds, but for battery-operated Arlos, that's too frequent and causes noticeable battery drain (I've seen 2-5% depletion per day). I've tested with the interval set to 1 hour, and it has been fine. Granted, that would not be optimal for reading property changes made outside of HASS (like updating video quality), but ideally functionality for controlling those settings would eventually get baked into this component so we don't have to rely so heavily on polling data.

tchellomello · 2018-05-31T19:36:08Z

@jwillaz I'm working on a PR that will change this behavior and get the data polling much better. I'm almost done with the new refactoring and I'll let you know as soon I get it published for testing.

MuppetOwl · 2018-06-01T07:50:57Z

Great news the Arlo component get some love :) I have a Arlo Q plus and would be great to be able to activate the motion detection from Home Assistant and also see the status for the sensors like wifi signal etc (They dont work on the Q now, just unknown status). Any plans to support more function for the Arlo Q? I can help if you need someone to test.

tchellomello · 2018-06-02T08:14:27Z

Guys, started some work at tchellomello/python-arlo#75 and potential changes to the HASS components here tchellomello/home-assistant@dev...arlo_fix

It's been working for me now but I want to leave it running for 1 or 2 days to see how it goes.

I'll keep you posted.

jwillaz · 2018-06-03T19:45:03Z

@tchellomello Mostly good, so far! I'm not seeing the timeouts that typically occur after several hours and the constant sensor warnings for taking too long. The one warning I receive on HASS startup is this:

Updating state for camera.xxxxxx (<class 'homeassistant.components.camera.arlo.ArloCam'>) took 1.890 seconds. Please report platform to the developers at https://goo.gl/Nvioub

Other than that, I am still seeing some steep battery drain on the cameras (2% in 12 hours). I checked queries from the Arlo hub in Pi-Hole, and it's sending out two queries every 2 minutes.

grantclem · 2018-06-04T02:02:20Z

Hi, just wondering how I go about testing this tchellomello/home-assistant@dev...arlo_fix? Custom_components?

grantclem · 2018-06-05T00:46:10Z

Hi, using Alok's repo I got it working, Few errors and warnings still coming up. But essentially working.

2018-06-04 23:22:33 WARNING (MainThread) [homeassistant.helpers.entity] Update of alarm_control_panel.base_station is taking over 10 seconds

2018-06-04 23:26:00 WARNING (MainThread) [homeassistant.components.alarm_control_panel] Updating arlo alarm_control_panel took longer than the scheduled update interval 0:00:30

2018-06-04 23:26:42 ERROR (MainThread) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/homeassistant/.homeassistant/custom_components/arlo.py", line 83, in hub_refresh
    update_base_station=True)
  File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/__init__.py", line 262, in update
    base.update()
  File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/base_station.py", line 457, in update
    self._attrs = self._session.refresh_attributes(self.name)
  File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/__init__.py", line 206, in refresh_attributes
    data = self.query(url).get('data')
AttributeError: 'NoneType' object has no attribute 'get'

tchellomello · 2018-06-05T02:38:47Z

@grantclem thanks for the report. This issue has been fixed and will be included in the next version.

2018-06-04 23:26:42 ERROR (MainThread) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/homeassistant/.homeassistant/custom_components/arlo.py", line 83, in hub_refresh
update_base_station=True)
File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/init.py", line 262, in update
base.update()
File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/base_station.py", line 457, in update
self._attrs = self._session.refresh_attributes(self.name)
File "/srv/homeassistant/lib/python3.6/site-packages/pyarlo/init.py", line 206, in refresh_attributes
data = self.query(url).get('data')
AttributeError: 'NoneType' object has no attribute 'get'

See: tchellomello/python-arlo#76

tchellomello · 2018-06-05T04:43:28Z

Guys, PR #14823 created. It's still flagged as WIP since I have to update the documentation but any feedback is very welcome.

Thanks

jurgenweber · 2018-06-12T06:05:06Z

woohoo! Does that mean it will be in 0.71.1 or 0.72.0?

Thanks

tchellomello · 2018-06-12T13:36:53Z

0.72.0

arsaboo mentioned this issue Mar 17, 2018

Arlo Q stopped working after newest update #13279

Closed

mwd102 mentioned this issue Mar 25, 2018

Modified arlo object to make smarter HTTP requests for an async envir… #13445

Closed

8 tasks

jurgenweber mentioned this issue Apr 21, 2018

Arlo Baby jeffreydwalter/arlo#60

Closed

tchellomello mentioned this issue Jun 2, 2018

Enhanced how Arlo queries API to reduce polls tchellomello/python-arlo#75

Merged

This was referenced Jun 5, 2018

Fixes AttributerError when requests failed to get a response tchellomello/python-arlo#76

Merged

Refactored Arlo component and enhanced Arlo API queries and times #14823

Merged

MartinHjelmare closed this as completed in #14823 Jun 12, 2018

home-assistant locked and limited conversation to collaborators Oct 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Miscellaneous Arlo warnings and errors #13176

Miscellaneous Arlo warnings and errors #13176

arsaboo commented Mar 13, 2018

tchellomello commented Mar 13, 2018

jwillaz commented Mar 13, 2018

jwillaz commented Mar 13, 2018

arsaboo commented Mar 17, 2018

JBenzie commented Mar 19, 2018

mwd102 commented Mar 25, 2018 •

edited

Loading

mwd102 commented Mar 25, 2018 •

edited

Loading

jwillaz commented Mar 27, 2018

mwd102 commented Mar 27, 2018 •

edited

Loading

jurgenweber commented Apr 10, 2018 •

edited

Loading

codypet commented Apr 17, 2018

jurgenweber commented Apr 19, 2018

jurgenweber commented Apr 21, 2018

jeffreydwalter commented Apr 21, 2018

arsaboo commented May 1, 2018

tchellomello commented May 1, 2018

rafuz commented May 20, 2018

jurgenweber commented May 21, 2018

tchellomello commented May 21, 2018

tchellomello commented May 29, 2018

jurgenweber commented May 29, 2018 •

edited

Loading

grantclem commented May 29, 2018

jwillaz commented May 31, 2018

tchellomello commented May 31, 2018

MuppetOwl commented Jun 1, 2018

tchellomello commented Jun 2, 2018 •

edited

Loading

jwillaz commented Jun 3, 2018

grantclem commented Jun 4, 2018

grantclem commented Jun 5, 2018 •

edited by MartinHjelmare

Loading

tchellomello commented Jun 5, 2018 •

edited

Loading

tchellomello commented Jun 5, 2018

jurgenweber commented Jun 12, 2018

tchellomello commented Jun 12, 2018

Miscellaneous Arlo warnings and errors #13176

Miscellaneous Arlo warnings and errors #13176

Comments

arsaboo commented Mar 13, 2018

tchellomello commented Mar 13, 2018

jwillaz commented Mar 13, 2018

jwillaz commented Mar 13, 2018

arsaboo commented Mar 17, 2018

JBenzie commented Mar 19, 2018

mwd102 commented Mar 25, 2018 • edited Loading

mwd102 commented Mar 25, 2018 • edited Loading

jwillaz commented Mar 27, 2018

mwd102 commented Mar 27, 2018 • edited Loading

jurgenweber commented Apr 10, 2018 • edited Loading

codypet commented Apr 17, 2018

jurgenweber commented Apr 19, 2018

jurgenweber commented Apr 21, 2018

jeffreydwalter commented Apr 21, 2018

arsaboo commented May 1, 2018

tchellomello commented May 1, 2018

rafuz commented May 20, 2018

jurgenweber commented May 21, 2018

tchellomello commented May 21, 2018

tchellomello commented May 29, 2018

jurgenweber commented May 29, 2018 • edited Loading

grantclem commented May 29, 2018

jwillaz commented May 31, 2018

tchellomello commented May 31, 2018

MuppetOwl commented Jun 1, 2018

tchellomello commented Jun 2, 2018 • edited Loading

jwillaz commented Jun 3, 2018

grantclem commented Jun 4, 2018

grantclem commented Jun 5, 2018 • edited by MartinHjelmare Loading

tchellomello commented Jun 5, 2018 • edited Loading

tchellomello commented Jun 5, 2018

jurgenweber commented Jun 12, 2018

tchellomello commented Jun 12, 2018

mwd102 commented Mar 25, 2018 •

edited

Loading

mwd102 commented Mar 25, 2018 •

edited

Loading

mwd102 commented Mar 27, 2018 •

edited

Loading

jurgenweber commented Apr 10, 2018 •

edited

Loading

jurgenweber commented May 29, 2018 •

edited

Loading

tchellomello commented Jun 2, 2018 •

edited

Loading

grantclem commented Jun 5, 2018 •

edited by MartinHjelmare

Loading

tchellomello commented Jun 5, 2018 •

edited

Loading