Skip to content

Commit a5f882e

Browse files
jbprattjbpratt
committed
kettle: migrate off of gs://kubernetes-jenkins
Switch the gs bucket referenced from `kubernetes-jenkins` -> `kubernetes-ci-logs` updates #33381 Signed-off-by: jbpratt <[email protected]>
1 parent 6a352c5 commit a5f882e

File tree

7 files changed

+62
-62
lines changed

7 files changed

+62
-62
lines changed

‎kettle/README.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ kubectl logs -l app=kettle
4040
PULLED 174
4141
ACK irrelevant 172
4242
EXTEND-ACK 2
43-
gs://kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/1136/kubeflow-presubmit/2385 True True 2018-07-06 07:51:49 PDT FAILED
44-
gs://kubernetes-jenkins/logs/ci-cri-containerd-e2e-ubuntu-gce/5742 True True 2018-07-06 07:44:17 PDT FAILURE
43+
gs://kubernetes-ci-logs/pr-logs/pull/kubeflow_kubeflow/1136/kubeflow-presubmit/2385 True True 2018-07-06 07:51:49 PDT FAILED
44+
gs://kubernetes-ci-logs/logs/ci-cri-containerd-e2e-ubuntu-gce/5742 True True 2018-07-06 07:44:17 PDT FAILURE
4545
ACK "finished.json" 2
4646
Downloading JUnit artifacts.
4747
```
@@ -102,9 +102,9 @@ A [postsubmit job](https://github.com/kubernetes/test-infra/blob/master/config/j
102102
103103
# PubSub
104104
105-
Kettle `stream.py` leverages Google Cloud [PubSub] to alert on GCS changes within the `kubernetes-jenkins` bucket. These events are tied to the `gcs-changes` Topic in the `kubernetes-jenkins` project where Prow job artifacts are collated. Each time an artifact is finalized, a PubSub event is triggered and Kettle collects job information when it sees a resource uploaded called `finished.json` (indicating the build completed).
105+
Kettle `stream.py` leverages Google Cloud [PubSub] to alert on GCS changes within the `kubernetes-ci-logs` bucket. These events are tied to the `gcs-changes` Topic in the `kubernetes-ci-logs` project where Prow job artifacts are collated. Each time an artifact is finalized, a PubSub event is triggered and Kettle collects job information when it sees a resource uploaded called `finished.json` (indicating the build completed).
106106

107-
[Topic Creation] can be performed by running `gcloud config set project kubernetes-jenkins` and `gsutil notification create -t gcs-changes -f json gs://kubernetes-jenkins`
107+
[Topic Creation] can be performed by running `gcloud config set project kubernetes-ci-logs` and `gsutil notification create -t gcs-changes -f json gs://kubernetes-ci-logs`
108108

109109
[Subscriptions] are in Kuberenetes Jenkins Build - PubSub.
110110
- kettle
@@ -114,14 +114,14 @@ They are split so that the staging instance does not consume events aimed at pro
114114

115115
These can be created via:
116116
```
117-
gcloud pubsub subscriptions create <subscription name> --topic=gcs-changes --topic-project="kubernetes-jenkins" --message-filter='attributes.eventType = "OBJECT_FINALIZE"'
117+
gcloud pubsub subscriptions create <subscription name> --topic=gcs-changes --topic-project="kubernetes-ci-logs" --message-filter='attributes.eventType = "OBJECT_FINALIZE"'
118118
```
119119

120120
### Auth
121121
For kettle to have permission, kettle's user needs access. When updating or changing a [Subscription] make sure to add `[email protected]` as a `PubSub Editor`.
122122
```
123123
gcloud pubsub subscriptions add-iam-policy-binding \
124-
projects/kubernetes-jenkins/subscriptions/kettle-staging \
124+
projects/kubernetes-ci-logs/subscriptions/kettle-staging \
125125
--member=serviceAccount:kettle@kubernetes-public.iam.gserviceaccount.com \
126126
--role=roles/pubsub.editor
127127
```
@@ -134,5 +134,5 @@ gcloud pubsub subscriptions add-iam-policy-binding \
134134
[Big Query All]: https://console.cloud.google.com/bigquery?project=kubernetes-public&page=table&t=all&d=build&p=kubernetes-public
135135
[Big Query Staging]: https://console.cloud.google.com/bigquery?project=kubernetes-public&page=table&t=staging&d=build&p=kubernetes-public
136136
[PubSub]: https://cloud.google.com/pubsub/docs
137-
[Subscriptions]: https://console.cloud.google.com/cloudpubsub/subscription/list?project=kubernetes-jenkins
137+
[Subscriptions]: https://console.cloud.google.com/cloudpubsub/subscription/list?project=kubernetes-ci-logs
138138
[Topic Creation]: https://cloud.google.com/storage/docs/reporting-changes#enabling

‎kettle/buckets.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
# collection phase, and defaults to true.
99
# exclude_jobs: A list of job names that will not be uploaded to BQ
1010

11-
gs://kubernetes-jenkins/logs/:
11+
gs://kubernetes-ci-logs/logs/:
1212
contact: "bentheelder"
1313
prefix: ""
1414
sequential: false

‎kettle/make_db.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -329,7 +329,7 @@ def main(db, jobs_dirs, threads, get_junit, build_limit, skip_gcs, client_class=
329329
"""Collect test info in matching jobs."""
330330
setup_logging()
331331
if not skip_gcs:
332-
get_all_builds(db, 'gs://kubernetes-jenkins/pr-logs', {'pr': True},
332+
get_all_builds(db, 'gs://kubernetes-ci-logs/pr-logs', {'pr': True},
333333
threads, client_class, build_limit)
334334
for bucket, metadata in jobs_dirs.items():
335335
if not bucket.endswith('/'):

‎kettle/make_db_test.py

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
static_epoch = 1641585162
2626

2727
TEST_BUCKETS_DATA = {
28-
'gs://kubernetes-jenkins/logs/': {'prefix': ''},
28+
'gs://kubernetes-ci-logs/logs/': {'prefix': ''},
2929
'gs://bucket1/': {'prefix': 'bucket1_prefix'},
3030
'gs://bucket2/': {'prefix': 'bucket2_prefix'}
3131
}
@@ -34,15 +34,15 @@
3434
class MockedClient(make_db.GCSClient):
3535
"""A GCSClient with stubs for external interactions."""
3636
NOW = static_epoch
37-
LOG_DIR = 'gs://kubernetes-jenkins/logs/'
37+
LOG_DIR = 'gs://kubernetes-ci-logs/logs/'
3838
JOB_DIR = LOG_DIR + 'fake/123/'
3939
ART_DIR = JOB_DIR + 'artifacts/'
4040
lists = {
4141
LOG_DIR: [LOG_DIR + 'fake/'],
4242
LOG_DIR + 'fake/': [JOB_DIR, LOG_DIR + 'fake/122/'],
4343
LOG_DIR + 'bad-latest/': [LOG_DIR + 'bad-latest/6/'],
4444
LOG_DIR + 'latest/': [LOG_DIR + 'latest/4/', LOG_DIR + 'latest/3/'],
45-
'gs://kubernetes-jenkins/pr-logs/directory/': [],
45+
'gs://kubernetes-ci-logs/pr-logs/directory/': [],
4646
ART_DIR: [ART_DIR + 'junit_01.xml'],
4747
ART_DIR.replace('123', '122'): [],
4848
}
@@ -76,7 +76,7 @@ class GCSClientTest(unittest.TestCase):
7676

7777
# pylint: disable=protected-access
7878

79-
JOBS_DIR = 'gs://kubernetes-jenkins/logs/'
79+
JOBS_DIR = 'gs://kubernetes-ci-logs/logs/'
8080

8181
def setUp(self):
8282
self.client = MockedClient(self.JOBS_DIR)
@@ -85,7 +85,7 @@ def test_get_junits(self):
8585
junits = self.client.get_junits_from_build(self.JOBS_DIR + 'fake/123')
8686
self.assertEqual(
8787
sorted(junits),
88-
['gs://kubernetes-jenkins/logs/fake/123/artifacts/junit_01.xml'])
88+
['gs://kubernetes-ci-logs/logs/fake/123/artifacts/junit_01.xml'])
8989

9090
def test_get_builds_normal_list(self):
9191
# normal case: lists a directory
@@ -184,14 +184,14 @@ def test_incremental_new(self):
184184

185185
class MockedClientNewer(MockedClient):
186186
NOW = static_epoch
187-
LOG_DIR = 'gs://kubernetes-jenkins/logs/'
187+
LOG_DIR = 'gs://kubernetes-ci-logs/logs/'
188188
JOB_DIR = LOG_DIR + 'fake/124/'
189189
ART_DIR = JOB_DIR + 'artifacts/'
190190
lists = {
191191
LOG_DIR: [LOG_DIR + 'fake/'],
192192
LOG_DIR + 'fake/': [JOB_DIR, LOG_DIR + 'fake/123/'],
193193
ART_DIR: [ART_DIR + 'junit_01.xml'],
194-
'gs://kubernetes-jenkins/pr-logs/directory/': [],
194+
'gs://kubernetes-ci-logs/pr-logs/directory/': [],
195195
}
196196
gets = {
197197
JOB_DIR + 'finished.json': {'timestamp': NOW},

‎kettle/make_json.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -115,9 +115,9 @@ def populate_path_to_job_and_number(self):
115115
if self.path.startswith(bucket):
116116
prefix = meta['prefix']
117117
break
118-
#if job path not in buckets.yaml or gs://kubernetes-jenkins/pr-logs it is unmatched
118+
#if job path not in buckets.yaml or gs://kubernetes-ci-logs/pr-logs it is unmatched
119119
else:
120-
if self.path.startswith('gs://kubernetes-jenkins/pr-logs'):
120+
if self.path.startswith('gs://kubernetes-ci-logs/pr-logs'):
121121
prefix = 'pr:'
122122
else:
123123
raise ValueError(f'unknown build path for {self.path} in known bucket paths')

‎kettle/make_json_test.py

Lines changed: 30 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ class ValidateBuckets(unittest.TestCase):
2929
def test_buckets(self):
3030
prefixes = set()
3131
for name, options in sorted(make_json.BUCKETS.items()):
32-
if name == 'gs://kubernetes-jenkins/logs/':
32+
if name == 'gs://kubernetes-ci-logs/logs/':
3333
continue # only bucket without a prefix
3434
prefix = options.get('prefix', '')
3535
self.assertNotEqual(prefix, '', 'bucket %s must have a prefix' % name)
@@ -46,8 +46,8 @@ class BuildObjectTests(unittest.TestCase):
4646
),
4747
(
4848
"Base_Build",
49-
make_json.Build("gs://kubernetes-jenkins/pr-logs/path", []),
50-
{"path":"gs://kubernetes-jenkins/pr-logs/path",
49+
make_json.Build("gs://kubernetes-ci-logs/pr-logs/path", []),
50+
{"path":"gs://kubernetes-ci-logs/pr-logs/path",
5151
"test": [],
5252
"tests_run": 0,
5353
"tests_failed": 0,
@@ -57,10 +57,10 @@ class BuildObjectTests(unittest.TestCase):
5757
(
5858
"Tests_populate",
5959
make_json.Build(
60-
"gs://kubernetes-jenkins/pr-logs/path",
60+
"gs://kubernetes-ci-logs/pr-logs/path",
6161
[{'name': 'Test1', 'failed': True}],
6262
),
63-
{"path":"gs://kubernetes-jenkins/pr-logs/path",
63+
{"path":"gs://kubernetes-ci-logs/pr-logs/path",
6464
"test": [{'name': 'Test1', 'failed': True}],
6565
"tests_run": 1,
6666
"tests_failed": 1,
@@ -143,9 +143,9 @@ def test_as_dict(self, _, build, expected):
143143
),
144144
])
145145
def test_populate_start(self, _, started, updates):
146-
build = make_json.Build("gs://kubernetes-jenkins/pr-logs/path", [])
146+
build = make_json.Build("gs://kubernetes-ci-logs/pr-logs/path", [])
147147
attrs = {
148-
"path":"gs://kubernetes-jenkins/pr-logs/path",
148+
"path":"gs://kubernetes-ci-logs/pr-logs/path",
149149
"test": [],
150150
"tests_run": 0,
151151
"tests_failed": 0,
@@ -223,8 +223,8 @@ def test_populate_start(self, _, started, updates):
223223
),
224224
])
225225
def test_populate_finish(self, _, finished, updates):
226-
build = make_json.Build("gs://kubernetes-jenkins/pr-logs/path", [])
227-
attrs = {"path":"gs://kubernetes-jenkins/pr-logs/path",
226+
build = make_json.Build("gs://kubernetes-ci-logs/pr-logs/path", [])
227+
attrs = {"path":"gs://kubernetes-ci-logs/pr-logs/path",
228228
"test": [],
229229
"tests_run": 0,
230230
"tests_failed": 0,
@@ -239,39 +239,39 @@ class GenerateBuilds(unittest.TestCase):
239239
@parameterized.expand([
240240
(
241241
"Basic_pass",
242-
"gs://kubernetes-jenkins/pr-logs/path",
242+
"gs://kubernetes-ci-logs/pr-logs/path",
243243
[{'name': "Test1", 'failed': False}],
244244
None,
245245
None,
246246
None,
247247
None,
248248
{
249249
'job': 'pr:pr-logs',
250-
'path': 'gs://kubernetes-jenkins/pr-logs/path',
250+
'path': 'gs://kubernetes-ci-logs/pr-logs/path',
251251
'test': [{'name': 'Test1', 'failed': False}],
252252
'tests_run': 1,
253253
'tests_failed':0,
254254
},
255255
),
256256
(
257257
"Basic_fail",
258-
"gs://kubernetes-jenkins/pr-logs/path",
258+
"gs://kubernetes-ci-logs/pr-logs/path",
259259
[{'name': "Test1", 'failed': True}],
260260
None,
261261
None,
262262
None,
263263
None,
264264
{
265265
'job': 'pr:pr-logs',
266-
'path': 'gs://kubernetes-jenkins/pr-logs/path',
266+
'path': 'gs://kubernetes-ci-logs/pr-logs/path',
267267
'test': [{'name': 'Test1', 'failed': True}],
268268
'tests_run': 1,
269269
'tests_failed':1,
270270
},
271271
),
272272
(
273273
"Ci_decorated",
274-
"gs://kubernetes-jenkins/pr-logs/path",
274+
"gs://kubernetes-ci-logs/pr-logs/path",
275275
[{'name': "Test1", 'failed': True}],
276276
{
277277
"timestamp":1595284709,
@@ -289,7 +289,7 @@ class GenerateBuilds(unittest.TestCase):
289289
None,
290290
{
291291
'job': 'pr:pr-logs',
292-
'path': 'gs://kubernetes-jenkins/pr-logs/path',
292+
'path': 'gs://kubernetes-ci-logs/pr-logs/path',
293293
'test': [{'name': 'Test1', 'failed': True}],
294294
'passed': True,
295295
'result': 'SUCCESS',
@@ -304,7 +304,7 @@ class GenerateBuilds(unittest.TestCase):
304304
),
305305
(
306306
"Pr_decorated",
307-
"gs://kubernetes-jenkins/pr-logs/path",
307+
"gs://kubernetes-ci-logs/pr-logs/path",
308308
[{'name': "Test1", 'failed': True}],
309309
{
310310
"timestamp":1595277241,
@@ -323,7 +323,7 @@ class GenerateBuilds(unittest.TestCase):
323323
None,
324324
{
325325
'job': 'pr:pr-logs',
326-
'path': 'gs://kubernetes-jenkins/pr-logs/path',
326+
'path': 'gs://kubernetes-ci-logs/pr-logs/path',
327327
'test': [{'name': 'Test1', 'failed': True}],
328328
'passed': True,
329329
'result': 'SUCCESS',
@@ -338,7 +338,7 @@ class GenerateBuilds(unittest.TestCase):
338338
),
339339
(
340340
"Pr_bootstrap",
341-
"gs://kubernetes-jenkins/pr-logs/path",
341+
"gs://kubernetes-ci-logs/pr-logs/path",
342342
[{'name': "Test1", 'failed': True}],
343343
{
344344
"node": "0790211c-cacb-11ea-a4b9-4a19d9b965b2",
@@ -371,7 +371,7 @@ class GenerateBuilds(unittest.TestCase):
371371
{
372372
'job': 'pr:pr-logs',
373373
'executor': '0790211c-cacb-11ea-a4b9-4a19d9b965b2',
374-
'path': 'gs://kubernetes-jenkins/pr-logs/path',
374+
'path': 'gs://kubernetes-ci-logs/pr-logs/path',
375375
'test': [{'name': 'Test1', 'failed': True}],
376376
'passed': True,
377377
'result': 'SUCCESS',
@@ -393,7 +393,7 @@ class GenerateBuilds(unittest.TestCase):
393393
),
394394
(
395395
"Ci_bootstrap",
396-
"gs://kubernetes-jenkins/pr-logs/path",
396+
"gs://kubernetes-ci-logs/pr-logs/path",
397397
[{'name': "Test1", 'failed': True}],
398398
{
399399
"timestamp":1595263104,
@@ -427,7 +427,7 @@ class GenerateBuilds(unittest.TestCase):
427427
{
428428
'job': 'pr:pr-logs',
429429
'executor': '592473ae-caa7-11ea-b130-525df2b76a8d',
430-
'path': 'gs://kubernetes-jenkins/pr-logs/path',
430+
'path': 'gs://kubernetes-ci-logs/pr-logs/path',
431431
'test': [{'name': 'Test1', 'failed': True}],
432432
'passed': True,
433433
'result': 'SUCCESS',
@@ -452,7 +452,7 @@ class GenerateBuilds(unittest.TestCase):
452452
),
453453
(
454454
"Started_no_meta_repo",
455-
"gs://kubernetes-jenkins/pr-logs/path",
455+
"gs://kubernetes-ci-logs/pr-logs/path",
456456
[{'name': "Test1", 'failed': False}],
457457
{
458458
"timestamp":1595263104,
@@ -469,7 +469,7 @@ class GenerateBuilds(unittest.TestCase):
469469
{
470470
'job': 'pr:pr-logs',
471471
'executor': '592473ae-caa7-11ea-b130-525df2b76a8d',
472-
'path': 'gs://kubernetes-jenkins/pr-logs/path',
472+
'path': 'gs://kubernetes-ci-logs/pr-logs/path',
473473
'test': [{'name': 'Test1', 'failed': False}],
474474
'tests_run': 1,
475475
'tests_failed':0,
@@ -495,9 +495,9 @@ def expect(path, job, number):
495495
build = make_json.Build(path, [])
496496
self.assertEqual((build.job, build.number), (job, number))
497497

498-
expect('gs://kubernetes-jenkins/logs/some-build/123', 'some-build', 123)
499-
expect('gs://kubernetes-jenkins/logs/some-build/123asdf', 'some-build', None)
500-
expect('gs://kubernetes-jenkins/pr-logs/123/e2e-node/456', 'pr:e2e-node', 456)
498+
expect('gs://kubernetes-ci-logs/logs/some-build/123', 'some-build', 123)
499+
expect('gs://kubernetes-ci-logs/logs/some-build/123asdf', 'some-build', None)
500+
expect('gs://kubernetes-ci-logs/pr-logs/123/e2e-node/456', 'pr:e2e-node', 456)
501501

502502
with self.assertRaises(ValueError):
503503
expect('gs://unknown-bucket/foo/123', None, None)
@@ -517,7 +517,7 @@ def expect(path, start, finish, results, **kwargs):
517517
row = make_json.row_for_build(path, start, finish, results)
518518
self.assertEqual(row, expected)
519519

520-
path = 'gs://kubernetes-jenkins/logs/J/123'
520+
path = 'gs://kubernetes-ci-logs/logs/J/123'
521521
expect(path, None, None, [], job='J', number=123)
522522
expect(path, None, None, [], job='J', number=123)
523523
expect(path,
@@ -559,7 +559,7 @@ def test_main(self):
559559
junits = ['<testsuite><testcase name="t1" time="3.0"></testcase></testsuite>']
560560

561561
def add_build(path, start, finish, result, junits):
562-
path = 'gs://kubernetes-jenkins/logs/%s' % path
562+
path = 'gs://kubernetes-ci-logs/logs/%s' % path
563563
self.db.insert_build(
564564
path, {'timestamp': start}, {'timestamp': finish, 'result': result})
565565
# fake build rowid doesn't matter here
@@ -607,8 +607,8 @@ def expect(args, needles, negneedles, expected_ret=None):
607607
expect([], [], ['123', '456', '457']) # reset only works for given day
608608

609609
# verify that direct paths work
610-
expect(['gs://kubernetes-jenkins/logs/some-job/123'], ['123'], [])
611-
expect(['gs://kubernetes-jenkins/logs/some-job/123'], ['123'], [])
610+
expect(['gs://kubernetes-ci-logs/logs/some-job/123'], ['123'], [])
611+
expect(['gs://kubernetes-ci-logs/logs/some-job/123'], ['123'], [])
612612

613613
# verify that assert_oldest works
614614
expect(['--days=30'], ['123', '456'], [])

0 commit comments

Comments
 (0)