Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

403 Forbidden during docker run #67

Open
moezubair opened this issue Jan 22, 2019 · 4 comments
Open

403 Forbidden during docker run #67

moezubair opened this issue Jan 22, 2019 · 4 comments

Comments

@moezubair
Copy link

Hey,
I am getting the following error when i run the docker image:

2019-01-22 16:56:43,455 DEBG 'druid-historical' stdout output:
2019-01-22T16:56:43,446 ERROR [ZkCoordinator] io.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=io.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z], segment=DataSegment{size=24664730, shardSpec=NoneShardSpec, metrics=[count, delta, variation, added, deleted], dimensions=[dma_code, continent_code, geo, area_code, robot, country_name, network, city, namespace, anonymous, unpatrolled, page, postal_code, language, newpage, user, region_lookup], version='2013-08-08T21:22:48.989Z', loadSpec={type=>s3_zip, bucket=>static.druid.io, key=>data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip}, interval=2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z, dataSource='wikipedia', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]
	at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:265) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [druid-services-0.12.1-selfcontained.jar:0.12.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.RuntimeException: org.jets3t.service.ServiceException: Request Error. -- ResponseCode: 403, ResponseStatus: Forbidden, RequestId: B11F1C837771B9B6, HostId: vVwwPXur/GyoYZ3wmK0pSa9gtYL06t5dtFJIqerGDqQwM7Rek/ABBrnbGJQtLCgiwDzzItlCIh0=
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:327) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
	at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	... 18 more
Caused by: org.jets3t.service.ServiceException: Request Error.
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:426) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]
	at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
	at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	... 18 more
Caused by: org.jets3t.service.impl.rest.HttpException: 403 Forbidden
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:425) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]
	at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) ~[?:?]
	at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
	at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
	... 18 more

2019-01-22 16:56:43,459 DEBG 'druid-coordinator' stdout output:
2019-01-22T16:56:43,458 INFO [main-EventThread] io.druid.server.coordinator.CuratorLoadQueuePeon - Server[/druid/loadQueue/172.17.0.2:8083] done processing [/druid/loadQueue/172.17.0.2:8083/wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]

2019-01-22 16:56:43,459 DEBG 'druid-historical' stdout output:
2019-01-22T16:56:43,458 INFO [ZkCoordinator] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]
2019-01-22T16:56:43,458 INFO [ZkCoordinator] io.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/172.17.0.2:8083/wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z] was removed
@nickwallen
Copy link

I am getting this also, but am not sure why. The stack trace seems to refer to retrieving data from an S3 bucket. The only use of S3 that I see here is in pulling down the example Wikipedia data.

@nickwallen
Copy link

The sample-data.sql has this ingest spec embedded in it. I assume the data set is not publicly available in S3?

{
  "dataSource": "wikipedia",
  "interval": "2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z",
  "version": "2013-08-08T21:22:48.989Z",
  "loadSpec": {
    "type": "s3_zip",
    "bucket": "static.druid.io",
    "key": "data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip"
  },
  "dimensions": "dma_code,continent_code,geo,area_code,robot,country_name,network,city,namespace,anonymous,unpatrolled,page,postal_code,language,newpage,user,region_lookup",
  "metrics": "count,delta,variation,added,deleted",
  "shardSpec": {
    "type": "none"
  },
  "binaryVersion": 9,
  "size": 24664730,
  "identifier": "wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z"
}

@moezubair
Copy link
Author

That seems to be the issue....the supervisor config does have S3 config in it.

what needs to be changed to load this from a different file maybe?

@moezubair
Copy link
Author

@nickwallen if you're still facing the problem, i found the fix.

The S3 cred info in supervisord.conf is no longer valid. I got the updated one from the druid main repo and it worked:

-Ddruid.s3.accessKey=AKIAJI7DG7CDECGBQ6NA -Ddruid.s3.secretKey=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

nickwallen added a commit to nickwallen/docker-druid that referenced this issue Jan 23, 2019
@nickwallen
Fix druid-io#67 403 Forbidden during docker run
a520781
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nickwallen @moezubair