-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Herd Uploader #370
Comments
Hi Tinshuk - The Uploader tool is included in our automated test suite in our environment so I believe it should not be too difficult to get working in your environment. And it's a good indication that the pre-registration worked. I would like to collect some information. But first -- I bet you already discovered the swagger docs that ship with each release. Actually I think this is in the CloudFormation output but so maybe you did not see it. But the docs are at /herd-app/docs/rest/index.html and they will help you with the Storages GET below and many other REST calls you will be making in the future! Please send:
I am also tagging @kenisteward here who can help troubleshoot. Thanks Tinshuk, Keni! |
Hi @nateiam, Please find details you asked,
|
When the uploader tries the actual upload, it uses the BDATA"s storage.directorypath to go to the actual s3 place. It looks like your storage doesn't have the attributes that tells where your s3 path is. If you could, try doing a stoarge put on the following attributes: {
"attributes": [
{
"name": "bucket.name",
"value": "yourBucketName"
}
]
} If this doesn't work let us know. We think this should fix it with minimal changes but there are other knobs we can tweak. |
I updated the storage with attributes as:
but getting same error as earlier I mentioned. |
Gotcha. Looks like you need to set the keyPrefix for the storage since you can't set the storage directory in the manifest.json. Maybe we can make that a feature of uploader? @nateiam
It looks like with the herd-uploader's manifest.json, you aren't actually allowed to specify the storage directory. Because of this, you'll have to setup the directory path via the storage's key.prefix.velocity.template. This can be any string. It also has replaceable values that are: S3 Key Prefix Velocity Template Examples:
|
@tinshuksingh Are you still having any issues? |
* commit '1d023e3d0c9a357c5ce0474c43a0b7172cfd74d6': DM-10691: FactoryCQ - UI - Migrate from Http to HttpClient DM-10691: FactoryCQ - UI - Migrate from Http to HttpClient DM-10691: FactoryCQ - UI - Migrate from Http to HttpClient
Hi Team,
We created
business object definition
and now trying to upload file to S3 bucket usingherd-uploader-0.63.0.jar
from ec2 instance.We followed the wiki steps which you mentioned to upload file to S3.
We are able to pre-register the
business object
with registration server successfully but after that gettingnull pointer exception
while reading the directory path.manifest.json
{
"namespace": "S3UploadNamespace",
"businessObjectDefinitionName": "S3BusinessDefination",
"businessObjectFormatUsage": "PRC",
"businessObjectFormatFileType": "TXT",
"businessObjectFormatVersion": "0",
"partitionKey": "PROCESS_DATE",
"partitionValue": "2014-04-01",
"storageName": "S3StorageUnit",
"subPartitionValues": [
"2014-04-01"
],
"manifestFiles" : [ {
"fileName" : "testFile1.gz",
"rowCount" : 0
}, {
"fileName" : "testFile2.gz",
"rowCount" : 0
} ]
}
We tried to pass directory path in
manifest.json
file but it was failing with exceptionUnrecognizedPropertyException
.manifest.json
{
"namespace": "S3UploadNamespace",
"businessObjectDefinitionName": "S3BusinessDefination",
"businessObjectFormatUsage": "PRC",
"businessObjectFormatFileType": "TXT",
"businessObjectFormatVersion": "0",
"partitionKey": "PROCESS_DATE",
"partitionValue": "2014-04-01",
"storageUnits": [ {
"storageName": "S3StorageUnit",
"storageDirectory": {
"directoryPath": "Herd_poc_bucket"
},
"storageFiles": [
{
"filePath": "testFile1.txt",
"fileSizeBytes": 0,
"rowCount": 0
}
],
"discoverStorageFiles": true
}],
"subPartitionValues": [
"2014-04-01"
],
"manifestFiles" : [ {
"fileName" : "testFile1.gz",
"rowCount" : 0
}, {
"fileName" : "testFile2.gz",
"rowCount" : 0
} ]
}
Please let us know if we are missing anything.
Thanks,
Tinshuk
The text was updated successfully, but these errors were encountered: