Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

K-14 list and set context (introduced configobj) #66

Merged
merged 29 commits into from
Nov 19, 2019
Merged

Conversation

ms-shankar
Copy link
Contributor

@ms-shankar ms-shankar commented Nov 4, 2019

Description

Closes issue 14 and issue 80

Checklist

  • An issue was first created before opening this pull request
  • The new code follows the kaos contribution guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests to ensure that prove my fix is effective or that my feature works

@ms-shankar ms-shankar changed the title [WIP]: K-14 list contexts using configobj K-14 list contexts using configobj Nov 7, 2019
@ms-shankar ms-shankar requested review from aalhour and jfri3d November 7, 2019 13:44
@ms-shankar ms-shankar marked this pull request as ready for review November 7, 2019 13:51
@ms-shankar ms-shankar changed the title K-14 list contexts using configobj K-14 list and set context (introduced configobj) Nov 7, 2019
Copy link
Contributor

@aalhour aalhour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential NameError runtime bug, string formatting and use of prints.

@aalhour
Copy link
Contributor

aalhour commented Nov 7, 2019

@ms-shankar - can you please commit and push your changes to the Pipfile and Pipfile.lock? It doesn't seem that the configobj library was added there.

@aalhour
Copy link
Contributor

aalhour commented Nov 7, 2019

User Testing Session (No. 1)

I performed a fresh install of kaos.

kaos is broken

The kaos command is broken due to Pipenv not installing the configobj dependency:

➜ kaos
Traceback (most recent call last):
  File "/Users/ahmad/.local/share/virtualenvs/kaos-azHSaT7n/bin/kaos", line 11, in <module>
    load_entry_point('kaos', 'console_scripts', 'kaos')()
  File "/Users/ahmad/.local/share/virtualenvs/kaos-azHSaT7n/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/ahmad/.local/share/virtualenvs/kaos-azHSaT7n/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/Users/ahmad/.local/share/virtualenvs/kaos-azHSaT7n/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/Users/ahmad/.local/share/virtualenvs/kaos-azHSaT7n/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/ahmad/Workspace/kaos/cli/kaos_cli/__init__.py", line 2, in <module>
    from kaos_cli.factories.simple_factory import SimpleFactory
  File "/Users/ahmad/Workspace/kaos/cli/kaos_cli/factories/simple_factory.py", line 1, in <module>
    from kaos_cli.facades.backend_facade import BackendFacade
  File "/Users/ahmad/Workspace/kaos/cli/kaos_cli/facades/backend_facade.py", line 11, in <module>
    from kaos_cli.services.state_service import StateService
  File "/Users/ahmad/Workspace/kaos/cli/kaos_cli/services/state_service.py", line 4, in <module>
    from configobj import ConfigObj
ModuleNotFoundError: No module named 'configobj'

kaos build list raises an internal TypeError after fresh install

I fixed the above problem by installing configobj manually (it should be reflected in the Pipfile and Pipfile.lock though) and then tried performing a kaos build list but it seems broken:

➜ kaos build list
Warning - Unknown error: TypeError("object of type 'NoneType' has no len()")

kaos build deploy -c DOCKER -vy raises an internal AttributeError after successful deployment

➜ kaos build deploy -c DOCKER -vy
...
Apply complete! Resources: 21 added, 0 changed, 0 destroyed.

Warning - Unknown error: AttributeError("'BackendFacade' object has no attribute 'create_context'")

kaos build list still raises an internal TypeError even after successful deploy

➜ kaos build deploy -c DOCKER -vy
...
Apply complete! Resources: 21 added, 0 changed, 0 destroyed.
Warning - Unknown error: AttributeError("'BackendFacade' object has no attribute 'create_context'")

➜ kaos build list
Warning - Unknown error: TypeError("object of type 'NoneType' has no len()")

kaos build set -c DOCKER raises an internal TypeError after successful deploy

➜ kaos build set -c DOCKER
Warning - Unknown error: TypeError("'NoneType' object is not iterable")

kaos destroy -c DOCKER -v[y] raises an internal KeyError

➜ kaos destroy -c DOCKER -vy
Warning - Unknown error: KeyError('DOCKER')

Still breaks even without passing the y argument:

➜ kaos destroy -c DOCKER -v
Warning - Are you sure about destroying kaos backend in DOCKER? [y/N]: y
Warning - Unknown error: KeyError('DOCKER')

EDIT: Discovered and added more issues

@aalhour
Copy link
Contributor

aalhour commented Nov 8, 2019

@ms-shankar - kaos build list doesn't break anymore, but it's not reporting my existing docker build, also the kaos destroy command is still broken, heck out the following shell output:

➜ kaos build list
Warning - No active builds found. Please run kaos build deploy to deploy an environment

➜ kaos build deploy -c DOCKER -vy
Aborting - kaos backend is already built.

➜ kaos destroy -c DOCKER -vy
Warning - Unknown error: KeyError('DOCKER')

@aalhour
Copy link
Contributor

aalhour commented Nov 8, 2019

Worked after force-build. You might want to include this in the CHANGELOG, a user out there might have to do a force-rebuild of their backend once they upgrade their kaos installation to your latest changes.

➜ kaos build deploy -c DOCKER -vyf
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Info - Endpoint successfully set to http://localhost:80/api/
Info - Successfully built kaos environment
➜ kaos build list
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+
➜ kaos destroy -c DOCKER -vy
Destroy complete! Resources: 21 destroyed.
Info - Successfully destroyed kaos environment
➜ kaos build list
Warning - No active builds found. Please run kaos build deploy to deploy an environment

@aalhour
Copy link
Contributor

aalhour commented Nov 8, 2019

Build backend on DOCKER:

➜ kaos build deploy -c DOCKER -vy
...

Apply complete! Resources: 21 added, 0 changed, 0 destroyed.

Info - Endpoint successfully set to http://localhost:80/api/
Info - Successfully built kaos environment

Get list of builds:

➜ kaos build list
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+

Deploy backend on AWS:

➜ kaos build deploy -c AWS -vy
...

Workspace "prod" already exists
Switched to workspace "prod".

...

module.registry.null_resource.build_push_tag[1]: Refreshing state... [id=676301818431185394]

Error: Failed to load config (; default context): invalid configuration: no configuration has been provided

  on ../../../../../../../Ahmad/Workspace/kaos/infrastructure/aws/envs/__working_prod/config.tf line 13, in provider "kubernetes":
  13: provider "kubernetes" {

...

module.registry.null_resource.build_push_tag[0]: Refreshing state... [id=3237394661645997731]
Error: Failed to load config (; default context): invalid configuration: no configuration has been provided
  on ../../../../../../../Ahmad/Workspace/kaos/infrastructure/aws/envs/__working_prod/config.tf line 13, in provider "kubernetes":
  13: provider "kubernetes" {

Warning - Unknown error: FileNotFoundError(2, 'No such file or directory')

@aalhour
Copy link
Contributor

aalhour commented Nov 13, 2019

There is some weird behavior going on with destroy now. After calling kaos destroy, the kaos build list command started returning duplicate entries:

➜ kaos build list
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+

Destroy the backend

➜ kaos destroy -c DOCKER -vy
...
Destroy complete! Resources: 21 destroyed.
Info - Successfully destroyed kaos environment

After successful destroy, kaos build list still reports the existence of two backends

➜ kaos build list
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+
|   1   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+

Try to destroy the backend again, but kaos destroy says backend doesn't exist

➜ kaos destroy -c DOCKER -vy
Warning - kaos backend in DOCKER has not been deployed!

Backend doesn't exist but kaos build list says otherwise

➜ kaos build list
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+
|   1   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+

Filesystem Hierarchy

➜ tree .kaos
.kaos
├── config
└── state
    ├── AWS
    │   └── None
    └── GCP
        └── None

Tried to force rebuild kaos

➜ kaos build deploy -c DOCKER -vfy
...
Apply complete! Resources: 21 added, 0 changed, 0 destroyed.

Info - Endpoint successfully set to http://localhost:80/api/
Info - Successfully built kaos environment

kaos buid list now reports back 3 backends

➜ kaos build list
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+
|   1   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+
|   2   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+

Filesystem Hierarchy

➜ tree .kaos
.kaos
├── config
└── state
    ├── AWS
    │   └── None
    ├── DOCKER
    │   ├── config.json
    │   └── terraform.tfstate
    └── GCP
        └── None

6 directories, 3 files

@ms-shankar
Copy link
Contributor Author

The earlier failure cases were due to incomplete deployments which were caused by stale infrastructure state configurations. The following test case scenarios were tested from my side (DOCKER + GCP stage):

(kaos) KI-3162 :: KAOS/1/kaos ‹k-14-list-contexts*› » kaos build list    
+-------+-----------+----------+-------+
| index |  context  | provider |  env  |
+-------+-----------+----------+-------+
|   0   | GCP_stage |   GCP    | stage |
+-------+-----------+----------+-------+
|   1   |   DOCKER  |  DOCKER  | local |
+-------+-----------+----------+-------+
(kaos) KI-3162 :: KAOS/1/kaos ‹k-14-list-contexts*› » kaos build active  
Info - Active context is - DOCKER
(kaos) KI-3162 :: KAOS/1/kaos ‹k-14-list-contexts*› » kaos build set -i 0
Info - Successfully set to context - GCP_stage
(kaos) KI-3162 :: KAOS/1/kaos ‹k-14-list-contexts*› » kaos build active  
Info - Active context is - GCP_stage
(kaos) KI-3162 :: KAOS/1/kaos ‹k-14-list-contexts*› » kaos destroy -c GCP -e stage -vy
Initializing modules...

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.
.
.
destroying <lines removed>
.
.
module.networking.google_compute_network.k8s-network: Still destroying... [id=kaos-2-stage-vnet, 10s elapsed]
module.networking.google_compute_network.k8s-network: Destruction complete after 17s

Destroy complete! Resources: 32 destroyed.
Info - Successfully destroyed kaos [stage] environment
(kaos) KI-3162 :: KAOS/1/kaos ‹k-14-list-contexts*› » kaos build active               
Info - Active context is - None
(kaos) KI-3162 :: KAOS/1/kaos ‹k-14-list-contexts*› » kaos build list  
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+
(kaos) KI-3162 :: KAOS/1/kaos ‹k-14-list-contexts*› » kaos build set -i 0
Info - Successfully set to context - DOCKER

@aalhour
Copy link
Contributor

aalhour commented Nov 15, 2019

Tried to trick kaos into thinking that it has an AWS_dev deployment, a KeyError gets triggered:

➜ tree .kaos
.kaos
├── config
└── state
    ├── AWS
    │   └── dev
    │       └── None
    └── DOCKER
        ├── config.json
        └── terraform.tfstate

4 directories, 4 files


➜ kaos destroy -c AWS
Warning - Are you sure about destroying kaos [prod] backend in AWS? [y/N]: y
Warning - kaos [prod] backend in AWS has not been deployed!


➜ kaos destroy -c AWS -e dev
Warning - Are you sure about destroying kaos [dev] backend in AWS? [y/N]: y
Warning - Unknown error: KeyError('AWS_dev')

Copy link
Contributor

@aalhour aalhour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kaos destroy command broke after pipenv update:

➜ kaos build list
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+


➜ tree .kaos
.kaos
├── config
└── state
    ├── AWS
    │   └── dev
    │       └── None
    └── DOCKER
        ├── config.json
        └── terraform.tfstate

4 directories, 4 files


➜ kaos destroy -c DOCKER -vy
Warning - Unknown error: KeyError('None')

@ms-shankar
Copy link
Contributor Author

ms-shankar commented Nov 18, 2019

The issue discovered above pertains to a failed deployment from the previous run. But while trying to simulate the reported issue another issue was discovered during destroy, which was:

(kaos) KI-3162 :: 1/2/kaos ‹k-14-list-contexts*› » kaos build destroy -c DOCKER -vy
Usage: kaos build [OPTIONS] COMMAND [ARGS]...
Try "kaos build --help" for help.

Error: No such command "destroy".
(kaos) KI-3162 :: 1/2/kaos ‹k-14-list-contexts*› » kaos destroy -c DOCKER -vy                                                                                                                                                     2 ↵
Initializing modules...

Initializing the backend...

Initializing provider plugins...

.
. <- More destroy lines ->

Destroy complete! Resources: 21 destroyed.
Warning - Unknown error: TypeError('sequence item 2: expected str instance, NoneType found')

kaos destroy logic has now been modified to handle empty env values as well. Tested OK

Copy link
Contributor

@aalhour aalhour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kaos build set fails with an uncaught error when there is no backend built

➜ kaos build list
Warning - No active builds found. Please run kaos build deploy to deploy an environment

➜ kaos build set -i 0
Warning - Unknown error: TypeError('sequence item 2: expected str instance, NoneType found')

Copy link
Contributor

@aalhour aalhour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kaos train deploy is broken after refactor:

➜ kaos build deploy -c DOCKER -vy
...
Apply complete! Resources: 21 added, 0 changed, 0 destroyed.

Info - Endpoint successfully set to http://localhost:80/api/
Info - Successfully built kaos environment


➜ kaos build list
+-------+---------+----------+-------+
| index | context | provider |  env  |
+-------+---------+----------+-------+
|   0   |  DOCKER |  DOCKER  | local |
+-------+---------+----------+-------+


➜ kaos workspace create -n mnist
Info - Successfully set mnist workspace


➜ kaos template get --name mnist
Info - Successfully loaded mnist template


➜ kaos train deploy -s templates/mnist/model-train \
                    -d templates/mnist/data
Warning - Missing endpoint! Please set with - kaos init

Copy link
Contributor

@aalhour aalhour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kaos workspace set is broken due to an uncaught exception

➜ kaos workspace list
+-----+-------+
| ind |  name |
+-----+-------+
|  0  | mnist |
+-----+-------+


➜ kaos workspace set -i 0
Warning - Unknown error: TypeError('list indices must be integers or slices, not str')

Copy link
Contributor

@aalhour aalhour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kaos workspace kill doesn't work anymore

➜ curl -X POST localhost/serve-mnist-ba1759/invocations \
         --data-binary @templates/mnist/test_payload.jpg
{"result":[3]}


➜ kaos workspace kill
Warning - Are you sure about killing all mnist resources? [y/N]: y
in delete
Warning - Unknown error: TypeError("remove() got an unexpected keyword argument 'workspace'")

@ms-shankar
Copy link
Contributor Author

The issue with workspace set by index is not due to the introduction of configobj added in this PR. However, this issue has now been resolved.

The second issue with the workspace kill is due to the introduction of configobj and has now been adjusted to work correctly.

Copy link
Contributor

@aalhour aalhour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-tested the Quickstart Guide, AWS Deployment as well as all previously discovered bugs; the issues are now solved. GCP deployments didn't go through but that has to do with some infrastructure issues that @ms-shankar has to look through.

LGTM.

@aalhour aalhour merged commit b6528ca into master Nov 19, 2019
@aalhour aalhour deleted the k-14-list-contexts branch November 19, 2019 15:08
ms-shankar pushed a commit that referenced this pull request Nov 27, 2019
) (#84)

* Update documentation with latest kaos build changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants