Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Service catalog install failed #182

Open
hostingnuggets opened this issue Apr 17, 2020 · 1 comment
Open

Service catalog install failed #182

hostingnuggets opened this issue Apr 17, 2020 · 1 comment

Comments

@hostingnuggets
Copy link

hostingnuggets commented Apr 17, 2020 •
edited
Loading

Hello,

I am install okd 3.11 with this script on a fresh CentOS 7.7 server and unfortunately the service catalog part of the deploy ansible script fails. Below I have pasted the two specific tasks which fail including the debug info.

The only custom settings I have for the installer script is that I have answered yes to use Let's Encrypt (the certbot part of the script went fine) and I am using export DISK="/dev/sdb".

Any hints or help is appreciated. Thanks in advance.

TASK [openshift_service_catalog : Wait for Controller Manager rollout success] *************************************************************************************************************************************************************************************************
FAILED - RETRYING: Wait for Controller Manager rollout success (5 retries left).
FAILED - RETRYING: Wait for Controller Manager rollout success (4 retries left).
FAILED - RETRYING: Wait for Controller Manager rollout success (3 retries left).
FAILED - RETRYING: Wait for Controller Manager rollout success (2 retries left).
FAILED - RETRYING: Wait for Controller Manager rollout success (1 retries left).
fatal: [<IP_REMOVED>]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["oc", "rollout", "status", "--config=/etc/origin/master/admin.kubeconfig", "-n", "kube-service-catalog", "ds/controller-manager"], "delta": "0:00:00.203854", "end": "2020-04-17 23:19:50.143786", "msg": "non-zero return code", "rc": 1, "start": "2020-04-17 23:19:49.939932", "stderr": "error: watch closed before Until timeout", "stderr_lines": ["error: watch closed before Until timeout"], "stdout": "Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...\nWaiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...", "stdout_lines": ["Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...", "Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available..."]}
...ignoring

TASK [openshift_service_catalog : Verify that the Catalog API Server is running] ***********************************************************************************************************************************************************************************************
FAILED - RETRYING: Verify that the Catalog API Server is running (60 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (59 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (58 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (57 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (56 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (55 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (54 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (53 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (52 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (51 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (50 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (49 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (48 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (47 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (46 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (45 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (44 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (43 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (42 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (41 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (40 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (39 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (38 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (37 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (36 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (35 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (34 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (33 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (32 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (31 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (30 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (29 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (28 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (27 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (26 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (25 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (24 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (23 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (22 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (21 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (20 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (19 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (18 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (17 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (16 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (15 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (14 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (13 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (12 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (11 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (10 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (9 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (8 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (7 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (6 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (5 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (4 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (3 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (2 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (1 retries left).
fatal: [<IP_REMOVED>]: FAILED! => {"attempts": 60, "changed": false, "cmd": ["curl", "-k", "https://apiserver.kube-service-catalog.svc/healthz"], "delta": "0:00:01.010253", "end": "2020-04-17 23:31:06.368636", "msg": "non-zero return code", "rc": 7, "start": "2020-04-17 23:31:05.358383", "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused", "stderr_lines": ["  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", "                                 Dload  Upload   Total   Spent    Left  Speed", "", "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [openshift_service_catalog : debug] ***************************************************************************************************************************************************************************************************************************************
ok: [<IP_REMOVED>] => {
    "msg": [
        "In project kube-service-catalog on server https://console-internal.mydomain.com:8443", 
        "", 
        "https://apiserver-kube-service-catalog.apps.mydomain.com (passthrough) to pod port secure (svc/apiserver)", 
        "  daemonset/apiserver manages docker.io/openshift/origin-service-catalog:v3.11", 
        "    generation #1 running for 31 minutes - 0/1 pods growing to 1", 
        "  pod/apiserver-hjbmt runs docker.io/openshift/origin-service-catalog:v3.11", 
        "", 
        "svc/controller-manager - 172.30.179.105:443 -> 6443", 
        "  daemonset/controller-manager manages docker.io/openshift/origin-service-catalog:v3.11", 
        "    generation #1 running for 31 minutes - 0/1 pods growing to 1", 
        "  pod/controller-manager-5479q runs docker.io/openshift/origin-service-catalog:v3.11", 
        "", 
        "Errors:", 
        "  * pod/controller-manager-5479q is crash-looping", 
        "", 
        "1 error, 1 warning identified, use 'oc status --suggest' to see details."
    ]
}

TASK [openshift_service_catalog : Get pods in the kube-service-catalog namespace] **********************************************************************************************************************************************************************************************
changed: [<IP_REMOVED>]

TASK [openshift_service_catalog : debug] ***************************************************************************************************************************************************************************************************************************************
ok: [<IP_REMOVED>] => {
    "msg": [
        "NAME                       READY     STATUS             RESTARTS   AGE       IP            NODE                     NOMINATED NODE", 
        "apiserver-hjbmt            0/1       Running            8          31m       10.128.0.32   myserver.mydomain.com   <none>", 
        "controller-manager-5479q   0/1       CrashLoopBackOff   11         31m       10.128.0.33   myserver.mydomain.com   <none>"
    ]
}

TASK [openshift_service_catalog : Get events in the kube-service-catalog namespace] ********************************************************************************************************************************************************************************************
changed: [<IP_REMOVED>]

TASK [openshift_service_catalog : debug] ***************************************************************************************************************************************************************************************************************************************
ok: [<IP_REMOVED>] => {
    "msg": [
        "LAST SEEN   FIRST SEEN   COUNT     NAME                                                  KIND        SUBOBJECT                             TYPE      REASON             SOURCE                               MESSAGE", 
        "31m         31m          1         apiserver.1606b708b62fc509                            DaemonSet                                         Normal    SuccessfulCreate   daemonset-controller                 Created pod: apiserver-hjbmt", 
        "31m         31m          1         apiserver-hjbmt.1606b70943ace0c8                      Pod         spec.containers{apiserver}            Normal    Pulling            kubelet, myserver.mydomain.com      pulling image \"docker.io/openshift/origin-service-catalog:v3.11\"", 
        "31m         31m          1         controller-manager.1606b709c9ea1b59                   DaemonSet                                         Normal    SuccessfulCreate   daemonset-controller                 Created pod: controller-manager-5479q", 
        "31m         31m          2         controller-manager-5479q.1606b709de00d7d6             Pod                                               Warning   FailedMount        kubelet, myserver.mydomain.com      MountVolume.SetUp failed for volume \"service-catalog-ssl\" : secrets \"controllermanager-ssl\" not found", 
        "31m         31m          1         apiserver-hjbmt.1606b70a9fb9cab4                      Pod         spec.containers{apiserver}            Normal    Pulled             kubelet, myserver.mydomain.com      Successfully pulled image \"docker.io/openshift/origin-service-catalog:v3.11\"", 
        "31m         31m          1         service-catalog-controller-manager.1606b70b77a69ae0   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "31m         31m          1         service-catalog-controller-manager.1606b70bda237e24   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "31m         31m          1         service-catalog-controller-manager.1606b70f719db525   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "30m         31m          4         controller-manager-5479q.1606b70b6b94652c             Pod         spec.containers{controller-manager}   Normal    Created            kubelet, myserver.mydomain.com      Created container", 
        "30m         31m          4         controller-manager-5479q.1606b70b769e9857             Pod         spec.containers{controller-manager}   Normal    Started            kubelet, myserver.mydomain.com      Started container", 
        "30m         30m          1         service-catalog-controller-manager.1606b7179303ccc6   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "29m         31m          5         controller-manager-5479q.1606b70b462e62a6             Pod         spec.containers{controller-manager}   Normal    Pulled             kubelet, myserver.mydomain.com      Container image \"docker.io/openshift/origin-service-catalog:v3.11\" already present on machine", 
        "29m         29m          1         service-catalog-controller-manager.1606b721912d165d   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "28m         31m          4         apiserver-hjbmt.1606b70fe7f2776a                      Pod         spec.containers{apiserver}            Normal    Pulled             kubelet, myserver.mydomain.com      Container image \"docker.io/openshift/origin-service-catalog:v3.11\" already present on machine", 
        "28m         31m          5         apiserver-hjbmt.1606b70acc234e25                      Pod         spec.containers{apiserver}            Normal    Created            kubelet, myserver.mydomain.com      Created container", 
        "28m         31m          5         apiserver-hjbmt.1606b70ada41daa4                      Pod         spec.containers{apiserver}            Normal    Started            kubelet, myserver.mydomain.com      Started container", 
        "28m         28m          1         service-catalog-controller-manager.1606b73739e3ccde   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "25m         25m          1         service-catalog-controller-manager.1606b75cbc43ac23   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "20m         20m          1         service-catalog-controller-manager.1606b7a3f3664094   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "16m         30m          61        apiserver-hjbmt.1606b7150294a040                      Pod         spec.containers{apiserver}            Warning   BackOff            kubelet, myserver.mydomain.com      Back-off restarting failed container", 
        "15m         15m          1         service-catalog-controller-manager.1606b7ebe9dfac99   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "10m         10m          1         service-catalog-controller-manager.1606b832ad46ce74   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "5m          5m           1         service-catalog-controller-manager.1606b879b2aab021   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "1m          11m          121       apiserver-hjbmt.1606b821a488e639                      Pod         spec.containers{apiserver}            Warning   Unhealthy          kubelet, myserver.mydomain.com      Readiness probe failed: HTTP probe failed with statuscode: 500", 
        "1m          31m          140       controller-manager-5479q.1606b70c2c1048b1             Pod         spec.containers{controller-manager}   Warning   BackOff            kubelet, myserver.mydomain.com      Back-off restarting failed container", 
        "15s         15s          1         service-catalog-controller-manager.1606b8c003642347   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader"
    ]
}

TASK [openshift_service_catalog : Get pod logs] ********************************************************************************************************************************************************************************************************************************
changed: [<IP_REMOVED>]

TASK [openshift_service_catalog : debug] ***************************************************************************************************************************************************************************************************************************************
ok: [<IP_REMOVED>] => {
    "msg": [
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45198]", 
        "I0417 21:30:52.331698       1 run_server.go:134] etcd checker called", 
        "I0417 21:30:53.267983       1 wrap.go:42] GET /healthz: (79.085µs) 200 [[kube-probe/1.11+] 10.128.0.1:45242]", 
        "E0417 21:30:53.331836       1 run_server.go:143] etcd failed to reach any server", 
        "I0417 21:30:53.332025       1 wrap.go:42] GET /healthz/ready: (1.003896742s) 500", 
        "goroutine 4841 [running]:", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420439d50, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420439d50, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc4204814c0, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac", 
        "net/http.Error(0x7f676e03dbb0, 0xc4200e6d60, 0xc4204a7000, 0x34, 0x1f4)", 
        "\t/usr/local/go/src/net/http/server.go:1959 +0xda", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:130 +0x4c4", 
        "net/http.HandlerFunc.ServeHTTP(0xc42022df40, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42088c9c0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x514", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1a820d2, 0x19, 0xc42062b290, 0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb5c0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc", 
        "net/http.HandlerFunc.ServeHTTP(0xc42026fef0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce4", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb600, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb640, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37a00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37a00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x178", 
        "net/http.HandlerFunc.ServeHTTP(0xc42008d0e0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37a00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4204c61c0, 0x1fa8700, 0xc4200e6d60, 0xc420a37a00, 0xc4201d9980)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d", 
        "created by github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5", 
        "", 
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45230]", 
        "I0417 21:30:57.327797       1 run_server.go:134] etcd checker called", 
        "E0417 21:30:58.328094       1 run_server.go:143] etcd failed to reach any server", 
        "I0417 21:30:58.328288       1 wrap.go:42] GET /healthz/ready: (1.000566658s) 500", 
        "goroutine 4861 [running]:", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420752000, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420752000, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc4203ec0e0, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac", 
        "net/http.Error(0x7f676e03dbb0, 0xc4209f2298, 0xc420b34ec0, 0x34, 0x1f4)", 
        "\t/usr/local/go/src/net/http/server.go:1959 +0xda", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:130 +0x4c4", 
        "net/http.HandlerFunc.ServeHTTP(0xc42022df40, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42088c9c0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x514", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42010f3b0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1a820d2, 0x19, 0xc42062b290, 0xc42010f3b0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb5c0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc", 
        "net/http.HandlerFunc.ServeHTTP(0xc42026fef0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce4", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb600, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb640, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x178", 
        "net/http.HandlerFunc.ServeHTTP(0xc42008d0e0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4204c61c0, 0x1fa8700, 0xc4209f2298, 0xc4200d4500, 0xc420bd7380)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d", 
        "created by github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5", 
        "", 
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45286]", 
        "I0417 21:31:02.327443       1 run_server.go:134] etcd checker called", 
        "I0417 21:31:03.271530       1 wrap.go:42] GET /healthz: (3.603614ms) 200 [[kube-probe/1.11+] 10.128.0.1:45368]", 
        "E0417 21:31:03.327854       1 run_server.go:143] etcd failed to reach any server", 
        "I0417 21:31:03.328053       1 wrap.go:42] GET /healthz/ready: (1.000644294s) 500", 
        "goroutine 4880 [running]:", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4202f1420, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4202f1420, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc420481ee0, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac", 
        "net/http.Error(0x7f676e03dbb0, 0xc4200e6e90, 0xc4204a7a00, 0x34, 0x1f4)", 
        "\t/usr/local/go/src/net/http/server.go:1959 +0xda", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:130 +0x4c4", 
        "net/http.HandlerFunc.ServeHTTP(0xc42022df40, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42088c9c0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x514", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1a820d2, 0x19, 0xc42062b290, 0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb5c0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc", 
        "net/http.HandlerFunc.ServeHTTP(0xc42026fef0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce4", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb600, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb640, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f400)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f400)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x178", 
        "net/http.HandlerFunc.ServeHTTP(0xc42008d0e0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f400)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4204c61c0, 0x1fa8700, 0xc4200e6e90, 0xc42002f400, 0xc4210a89c0)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d", 
        "created by github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5", 
        "", 
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45356]", 
        "I0417 21:31:07.329046       1 run_server.go:134] etcd checker called", 
        "E0417 21:31:08.329250       1 run_server.go:143] etcd failed to reach any server", 
        "I0417 21:31:08.329370       1 wrap.go:42] GET /healthz/ready: (1.001460942s) 500", 
        "goroutine 4948 [running]:", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4202f1c00, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4202f1c00, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc420510a20, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac", 
        "net/http.Error(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420bc0fc0, 0x34, 0x1f4)", 
        "\t/usr/local/go/src/net/http/server.go:1959 +0xda", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:130 +0x4c4", 
        "net/http.HandlerFunc.ServeHTTP(0xc42022df40, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42088c9c0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x514", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1a820d2, 0x19, 0xc42062b290, 0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb5c0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc", 
        "net/http.HandlerFunc.ServeHTTP(0xc42026fef0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce4", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb600, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb640, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16200)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16200)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x178", 
        "net/http.HandlerFunc.ServeHTTP(0xc42008d0e0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16200)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4204c61c0, 0x1fa8700, 0xc4200e6ff8, 0xc420d16200, 0xc4210a92c0)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d", 
        "created by github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5", 
        "", 
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45422]"
    ]
}

TASK [openshift_service_catalog : Report errors] *******************************************************************************************************************************************************************************************************************************
fatal: [<IP_REMOVED>]: FAILED! => {"changed": false, "msg": "Catalog install failed."}
	to retry, use: --limit @/home/user/installcentos/openshift-ansible/playbooks/deploy_cluster.retry

The /var/log/messages log file contains these errors repeated every 5 seconds:


Apr 18 11:19:30 myserver origin-node: I0418 11:19:30.480421    2281 prober.go:111] Readiness probe for "apiserver-66bsx_kube-service-catalog(819aff89-8150-11ea-89a7-a0369fa3e320):apiserver" failed (failure): HTTP probe failed with statuscode: 500
Apr 18 11:19:30 myserver origin-node: I0418 11:19:30.995262    2281 prober.go:111] Readiness probe for "controller-manager-jpssz_kube-service-catalog(818589f3-8150-11ea-89a7-a0369fa3e320):controller-manager" failed (failure): HTTP probe failed with statuscode: 500
Apr 18 11:19:35 myserver origin-node: I0418 11:19:35.483845    2281 prober.go:111] Readiness probe for "apiserver-66bsx_kube-service-catalog(819aff89-8150-11ea-89a7-a0369fa3e320):apiserver" failed (failure): HTTP probe failed with statuscode: 500
Apr 18 11:19:35 myserver origin-node: I0418 11:19:35.993962    2281 prober.go:111] Readiness probe for "controller-manager-jpssz_kube-service-catalog(818589f3-8150-11ea-89a7-a0369fa3e320):controller-manager" failed (failure): HTTP probe failed with statuscode: 500

Then the /usr/local/bin/master-logs api api command shows the following errors repeated also quite often:

E0418 11:15:33.399732       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:15:43.417797       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:15:53.436591       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:03.455844       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:13.474989       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:23.493941       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:33.512922       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:43.531735       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:53.551345       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:17:03.569631       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:17:13.596194       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:17:23.615032       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
I0418 11:17:27.656105       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0418 11:17:27.658974       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0418 11:17:27.658985       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0418 11:17:28.650920       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.servicecatalog.k8s.io
E0418 11:17:28.651017       1 controller.go:111] loading OpenAPI spec for "v1beta1.servicecatalog.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0418 11:17:28.651025       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.servicecatalog.k8s.io: Rate Limited Requeue.
@hostingnuggets
Copy link
Author

I re-installed the whole server again and now it went through the service catalog... I can't really explain why. Leaving this issue open for a few days just for reference and in case any one knows what that was about.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hostingnuggets