-
Notifications
You must be signed in to change notification settings - Fork 3
/
HOWTO.txt
333 lines (245 loc) · 10.9 KB
/
HOWTO.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
===============================
Howto for developers of ConPaaS
===============================
--------------------
Source code location
--------------------
The official repository is hosted in github:
https://github.com/ConPaaS-team/conpaas
---------------------------------
How to compile and build packages
---------------------------------
$CPS_ROOT/mkdist.sh 1.4.x
will build 5 tarballs:
cpsclient-1.4.x.tar.gz
cpsdirector-1.4.x.tar.gz
cpsfrontend-1.4.x.tar.gz
cpslib-1.4.x.tar.gz
cps-tools-1.4.x.tar.gz
---------------------
How to make a release
---------------------
Checklist:
- create AMIs if necessary
- update AMIs in director.cfg.example
- update AMIs in documentation file docs/installation.rst , section "Pre-built Amazon Machine Images"
- travis test OK
- functional tests OK on OpenNebula and EC2? on other clouds?
- test on Amazon with director with a public address
- check that new features have documentation, and other features' documentation is up to date, check for typos
- mkdist $new_version_id
- update doc references to ConPaaS version
sed -i 's/1.3.0/1.3.1/' docs/conf.py
sed -i 's/1.3.0/1.3.1/' docs/installation.rst
- svn commit # commit the version number
- tag the version: svn copy svn+ssh://[email protected]/svnroot/contrail/trunk/conpaas svn+ssh://[email protected]/svnroot/contrail/tags/conpaas-1.3.1
- upload the 4 tarballs to www.conpaas.eu/dl/
- update the links to readthedocs inside the iframe in the www.conpaas.eu/documentation/{user-guide,installation-guide,internals-guide} to point to http://conpaas-team.readthedocs.org/en/1.3.1/
- make the announcement with the release notes
------------------
Running unit tests
------------------
Locally
=======
$CPS_ROOT/conpaas-services/src/tests/unit-tests.sh
$CPS_ROOT/conpaas-director/unit-tests.sh
$CPS_ROOT/cps-tools/unit-tests.sh
On Travis
=========
In https://github.com/ConPaaS-team/conpaas, there is a hook to Travis continuous integration testing.
https://travis-ci.org/ConPaaS-team/conpaas
Each time there is a push in Github ConPaaS-team/conpaas, Travis notified, clone the git repo and run the script travis.sh as specified in file .travis.yml
$CPS_ROOT/.travis.yml
$CPS_ROOT/travis.sh
There is a command line interface to travis:
https://github.com/travis-ci/travis
when installed you can easily retrieve the LARGE log (hard with browsers).
travis login
> enter your username and your password of Github
Then get the last log:
travis logs -r ConPaaS-team/conpaas > travis.log
------------------------
Code source architecture
------------------------
Directories in $CPS_ROOT:
conpaas-blueprints
unused
conpaas-client
command line client cpsclient.py, depends on conpaas-services
conpaas-director
conpaas director, depends on conpaas-services
conpaas-director/functional-tests
some scripts using the cpsclient.py command line to test a full installation of ConPaaS with real clouds
conpaas-frontend
web frontend GUI, depends on conpaas-services
conpaas-services
main ConPaaS code including common modules, manager codes, and agent codes.
cps-tools
command line client cps-tools, depends on conpaas-services
docs
ConPaaS documentation
examples
example applications that can be deployed in ConPaaS
-----------
Developping
-----------
Using Eclipse
* with Pydev:
- pep8 support: Preferences > PyDev > Editor > Code Analysis > tab 'pep8.py': add '--ignore=E501' to Arguments to skip too-long-line warnings
WARNING: see PEP8 section below
- pylint support: Preferences > PyDev > PyLint: check 'Use PyLint?' however issues related to PYTHONPATH setting for pylint
* Egit plugin for git support
* subversion plugin
Using PyCharm
----
PEP8
----
ConPaaS Python code syntax is not consistent.
Better thing is to comply to PEP8 official syntax recommendation.
The command line 'pep8' can spot Python code that does not comply to PEP8.
However, PEP8 line length of 80 characters is the hardest rule.
Better disable line-length rule first: pep8 --ignore=E501 mycode.py
Some part of ConPaaS code is already complying to PEP8 without line-length rule
* $CPS_ROOT/conpaas-services/src/conpaas/services/webservers/*
* $CPS_ROOT/conpaas-services/src/conpaas/services/mysql/*
It is enforced by the $CPS_ROOT/conpaas-services/src/tests/unit-tests.sh script to prevent any regression regarding PEP8.
Do not manually fix existing code to comply to PEP8, there is a script to do that:
autopep8 --in-place --ignore=E501 mycode.py
by ignoring error E501 (line length), autopep8 will not try to fold long lines on several lines, but will still fix other syntax errors.
------
PYLINT
------
Pylint is a static code analyzer for Python.
It helps detect some errors before running a Python script, however it is quite limited because Python is a un-typed language and quite dynamic.
Some ConPaaS code is passing Pylint:
* $CPS_ROOT/conpaas-services/src/conpaas/services/webservers/*
* $CPS_ROOT/conpaas-services/src/conpaas/services/mysql/*
It is enforced by the $CPS_ROOT/conpaas-services/src/tests/unit-tests.sh script to prevent any regression regarding Pylint.
----------------------------
How to generate a disk image
----------------------------
see $CPS_ROOT/conpaas-services/scripts/create_vm/README
------------------------------
Howto run ConPaaS on GRID 5000
------------------------------
see $CPS_ROOT/grid5000/README
--------------------------
How to extract ConPaaS API
--------------------------
see $CPS_ROOT/cps-api/README
--------------------------------
How to run ConPaaS on Amazon EC2
--------------------------------
Use amazon command line tools:
* Get credentials from Amazon
* follow http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SettingUp_CommandLine.html
unzip -x ec2-api-tools.zip -d $HOME
In $HOME/.profile
------------------------
# Set ec2-tools env vars
export JAVA_HOME="/usr/lib/jvm/java-6-openjdk-amd64/jre"
export EC2_HOME="/home/$USER/ec2-api-tools-1.6.11.0"
export PATH="$PATH:$EC2_HOME/bin"
export AWS_ACCESS_KEY=...
export AWS_SECRET_KEY=...
# for the register-image-ec2-*.sh scripts
export EC2_ACCESS_KEY="$AWS_ACCESS_KEY"
export EC2_SECRET_KEY="$AWS_SECRET_KEY"
------------------------
List amazon things:
ec2-describe-regions
ec2-describe-instances
ec2-describe-volumes
ec2-describe-snapshots
Amazon S3 buckets
https://console.aws.amazon.com/s3
Upload a new image and create a new AMI
sudo apt-get install ec2-ami-tools
* Must create an EC2 instance, upload the image, and upload the script, then run the script on the image.
- Created a micro instance using the conpaas-1.3.0 image, added a new 10Gb volume in EBS as /dev/sdb, pwd contrail for root
- ssh root@ec2....
apt-get install rsync
mktemp -d
mkfs.ext3 /dev/xvdb
mount /dev/xvdb /tmp/tmp.xxxxxxxxx
ls /tmp/tmp.xxxxxxxxx
df -h /tmp/tmp.xxxxxxxxx
- local$:
rsync -aS conpaas.img root@ec2...:/tmp/tmp.xxxxxxxx/conpaas.img
scp register-image-ec2-* *.cfg 40_custom root@ec2...:/tmp/tmp.xxxxxxxx/
- ssh root@ec2$
cd /tmp/tmp.xxxxxxxx/
./register-image-ec2-ebs.sh conpaas.img
- make created AMI public
* Some of my public AMIs are not visible in the list of public AMI in region Oregon (us-west-2):
I can see them as the owner, and they are explicitly public:
$ ec2-describe-images -U https://ec2.us-west-2.amazonaws.com -o 956027799375 | egrep '(ami-cc950efc|ami-f4c75fc4|ami-60940f50|ami-60940f50)'
IMAGE ami-f4c75fc4 956027799375/conpaas-1.3.0-2 956027799375 available public x86_64 machine aki-fc37bacc ebs paravirtual xen
IMAGE ami-cc950efc 956027799375/conpaas-1.3.0-2.1 956027799375 available public x86_64 machine aki-fc37bacc ebs paravirtual xen
IMAGE ami-60940f50 956027799375/conpaas-1.3.0-2.2 956027799375 available public x86_64 machine aki-fc37bacc ebs paravirtual xen
However, they do not appear from in the list of publicly available AMIs:
$ ec2-describe-images -U https://ec2.us-west-2.amazonaws.com -x all | egrep '(ami-cc950efc|ami-f4c75fc4|ami-60940f50)'
$
These AMIs were copied from other regions, two through the AWS web console interface:
ami-b79271c0 in eu-west-1 (Ireland) -----copied to Virginia -----> ami-c3045aaa -----copied to Oregon --------> ami-f4c75fc4
ami-b79271c0 in eu-west-1 (Ireland) ---------------------------copied to Oregon---------------------------> ami-cc950efc
and one AMI copied from command line:
ami-b79271c0 in eu-west-1 (Ireland) ---------------------------copied to Oregon ---------------------------> ami-60940f50
command was
$ ec2cpimg -U https://ec2.us-west-2.amazonaws.com -r eu-west-1 -s ami-b79271c0 -n conpaas-1.3.0-2.2 -d "Copied from eu-west-1.ami-b79271c0 using ec2cpimg command line."
IMAGE ami-60940f50
My issue seems related to a previous thread:
[https://forums.aws.amazon.com/thread.jspa?messageID=454705]
however it remained unsolved since May-July 2013....
I've also tried to copy from Ireland to North Virgnia again (ami-7f163316) but it does not appear in the list of public AMIs... so it is not specific to Oregon.
SOLUTION:
## make private
ec2-modify-image-attribute -U https://ec2.us-west-2.amazonaws.com ami-cc950efc -l -r all
## yes, this step seems required
ec2-describe-images -U https://ec2.us-west-2.amazonaws.com -o 956027799375 | egrep '(ami-cc950efc|ami-f4c75fc4|ami-60940f50)'
## make public
ec2-modify-image-attribute -U https://ec2.us-west-2.amazonaws.com ami-cc950efc -l -a all
## checking that it appears in the public list
ec2-describe-images -U https://ec2.us-west-2.amazonaws.com -x all | egrep '(ami-cc950efc|ami-f4c75fc4|ami-60940f50)'
----------------------------------
how to clean the director database
----------------------------------
sudo sqlite3 /etc/cpsdirector/director.db
> delete from resource;
> delete from service;
> delete from application;
------------------
Log file locations
------------------
On the director:
/var/log/apache2/director-error.log
/var/log/cpsdirector/debugging.log
On a manager:
/root/manager.err
/root/manager.out
/var/log/cpsmanager.log # the application manager's log
/var/log/cpsmanager{1,2,3,...}.log # the service manager's logs
From client:
cps-tools application get_log <aid> # the application manager's log
cps-tools service get_log <aid> <sid> # the service manager's logs
On an agent:
/root/agent.err
/root/agent.out
/var/log/cpsagent.log
cps-tools service get_agent_log <aid> <sid> <agent_id>
For web frontend:
/var/log/apache2/error.log
Javascript console in browser
On mysql agents:
/root/mysql.log
==========================
==========================
* What is missing? (cf conpaas-todo.odt)
- more unit tests, more integration tests
- a continuous integration test system
- a class diagram
- sequence diagrams, at least for the generic work of director-managers-agents relations
- more documentation in the API
- separate test code from function code (too many main in py files)
- more generic code (abstracted in BaseManager and BaseAgent)
- a generic load tester: useful for tests and demos