Selenium tests

[tomas-edwardsson]

Adds support for selenium browser based testing of /status

The tests are skipped if selenium is not installed.

[titilambert]

Nice :)

[palli]

Is there a way to get these tests up and running inside travis ? I worry otherwise these tests get merged in, and then never used.

Failing that, can we make a service on status.adagios.org that tests against git master ?

[pall-valmundsson]

It should be. https://saucelabs.com/opensauce https://docs.saucelabs.com/ci-integrations/travis-ci/

[pall-valmundsson]

One caveat, it doesn't work for pull requests. At least not for PRs not originating from the upstream repo, http://docs.travis-ci.com/user/sauce-connect/ (first paragraph) and http://docs.travis-ci.com/user/pull-requests/#Security-Restrictions-when-testing-Pull-Requests.

It has to do with exposing the secure variables.

[palli]

Well if there is nothing we can do about it... Then so be it :)

Having master branch tested is better than nothing.
On Oct 24, 2014 11:33 AM, "Pall Valmundsson" [email protected]
wrote:

One caveat, it doesn't work for pull requests. At least not for PRs not
originating from the upstream repo,
http://docs.travis-ci.com/user/sauce-connect/ (first paragraph) and
http://docs.travis-ci.com/user/pull-requests/#Security-Restrictions-when-testing-Pull-Requests
.

It has to do with exposing the secure variables.

—
Reply to this email directly or view it on GitHub
#441 (comment).

[pall-valmundsson]

As a workaround external PRs can be manually merged into new branches locally which are then pushed and tested as "upstream branches". A bit of work but might prove helpful for PRs deemed potentially dangerous, the PR submitter can then add patches to the new upstream branch if needed.

[titilambert]

Hello !
@palli @pall-valmundsson
We (Savoir-faire Linux) can host selenium tests on our public jenkins, if you want to.

[palli]

@tomas-edwardsson can you add a selenium test that covers the problem discovered in #440 and #433 ?

[tomas-edwardsson]

@palli added objectbrowser tests and integrated with Saucelabs, https://saucelabs.com/u/tommi

[titilambert]

@tomas-edwardsson nice job !

[palli]

This is really useful.

Just a quick question. Does this test return a false green if we fix #433 in such a way that the objectbrowser window shows errors instead of hanging when there is a problem ?

(i.e. should the test look for 'localhost' or something similar in the host table)

[tomas-edwardsson]

The test counts rows in the host-table tbody. If we setup a error handler we will not add a row to the host-table containing the error but a div similar to the one that shows Loading. Therefore it should flag as failed.

Here is the output after I intentionally broke adagios.js

======================================================================
FAIL: test_contacts_loading (adagios.objectbrowser.tests.SeleniumObjectBrowserTestCase)
Test if contacts under configure loads
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tommi/coding/adagios/adagios/../adagios/objectbrowser/tests.py", line 552, in test_contacts_loading
    self.assertTrue(False, "Timed out waiting for contact table to load")
AssertionError: Timed out waiting for contact table to load

======================================================================
FAIL: test_hosts_loading (adagios.objectbrowser.tests.SeleniumObjectBrowserTestCase)
Test if hosts under configure loads
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tommi/coding/adagios/adagios/../adagios/objectbrowser/tests.py", line 571, in test_hosts_loading
    self.assertTrue(False, "Timed out waiting for host table to load")
AssertionError: Timed out waiting for host table to load

----------------------------------------------------------------------

Breakage

diff --git a/adagios/media/js/adagios.js b/adagios/media/js/adagios.js
index a5ffc92..9ea938f 100644
--- a/adagios/media/js/adagios.js
+++ b/adagios/media/js/adagios.js
@@ -230,7 +230,7 @@ Generated by coffee-script, any changes should be made to the adagios.coffee fil
           if ("cAltName" in field) json_query_fields.push(field["cAltName"]);
           if ("cHidden" in field) return json_query_fields.push(field["cHidden"]);
         });
-        $.getJSON("../rest/pynag/json/get_objects", {
+        $.getJSON("../rest/pynag/json/get_objectas", {
           object_type: object_type,
           with_fields: json_query_fields.join(",")
         }, function(data) {

[palli]

Ok merging this as is.

If the test case handles errors, then you should consider renaming the error message (current error message indicates that a timeout occured)

[tomas-edwardsson]

Which is correct, it is a timeout error. When loading asynchronous content, I put in a wait for load. If the code is slow or does not load the asynchronous content at all, it is a timeout error.

[palli]

I think its a stretch to talk about a timeout error in cases where you have asynchronous call with no error callbacks.

Just think of the use case when this test breaks, and a helpful developers wants to see why the test is broken. Will the troubleshooter be looking at why the backend is serving errors, or will he be trying to profile backend use cases and/or retrying the unit tests with higher timeouts in the hopes that this was a fluke/slowness, in Travis ?

[palli]

Can we do this without having a global variable ?