Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EC2 User Data Limit Exceeded #393

Open
stevejhall opened this issue Oct 29, 2018 · 7 comments
Open

EC2 User Data Limit Exceeded #393

stevejhall opened this issue Oct 29, 2018 · 7 comments

Comments

@stevejhall
Copy link

When following the demo install shown here: https://github.com/FINRAOS/herd/wiki/demo-install

It appears that the user data exceeds an AWS limit. This is the error I got:
The following resource(s) failed to create: [herdApplicationServer]. . Rollback requested by user.
CREATE_FAILED | AWS::EC2::Instance | herdApplicationServer | User data is limited to 16384 bytes (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 3a...)

A bit of Googling seems to indicate that AWS has an upper limit of 16K

By way of troubleshooting I hacked out a big chunk of the UserData, just to see if I could get past this error. That test was successful. However, I do not fully understand the impacts of doing so.

This is what I deleted
"psql -f herd.postgres.0.01.0-to-0.02.0.upgrade.sql\n",
"psql -f herd.postgres.0.02.0-to-0.03.0.upgrade.sql\n",
"psql -f herd.postgres.0.03.0-to-0.04.0.upgrade.sql\n",
"psql -f herd.postgres.0.04.0-to-0.05.0.upgrade.sql\n",
"psql -f herd.postgres.0.05.0-to-0.06.0.upgrade.sql\n",
"psql -f herd.postgres.0.06.0-to-0.07.0.upgrade.sql\n",
"psql -f herd.postgres.0.07.0-to-0.08.0.upgrade.sql\n",
"psql -f herd.postgres.0.08.0-to-0.09.0.upgrade.sql\n",
"psql -f herd.postgres.0.09.0-to-0.10.0.upgrade.sql\n",
"psql -f herd.postgres.0.10.0-to-0.11.0.upgrade.sql\n",
"psql -f herd.postgres.0.11.0-to-0.12.0.upgrade.sql\n",
"psql -f herd.postgres.0.12.0-to-0.13.0.upgrade.sql\n",
"psql -f herd.postgres.0.13.0-to-0.14.0.upgrade.sql\n",
"psql -f herd.postgres.0.14.0-to-0.15.0.upgrade.sql\n",
"psql -f herd.postgres.0.15.0-to-0.16.0.upgrade.sql\n",
"psql -f herd.postgres.0.16.0-to-0.17.0.upgrade.sql\n",
"psql -f herd.postgres.0.17.0-to-0.18.0.upgrade.sql\n",
"psql -f herd.postgres.0.18.0-to-0.19.0.upgrade.sql\n",
"psql -f herd.postgres.0.19.0-to-0.20.0.upgrade.sql\n",
"psql -f herd.postgres.0.20.0-to-0.21.0.upgrade.sql\n",
"psql -f herd.postgres.0.21.0-to-0.22.0.upgrade.sql\n",
"psql -f herd.postgres.0.22.0-to-0.23.0.upgrade.sql\n",
"psql -f herd.postgres.0.23.0-to-0.24.0.upgrade.sql\n",
"psql -f herd.postgres.0.24.0-to-0.25.0.upgrade.sql\n",
"psql -f herd.postgres.0.25.0-to-0.26.0.upgrade.sql\n",
"psql -f herd.postgres.0.26.0-to-0.27.0.upgrade.sql\n",
"psql -f herd.postgres.0.27.0-to-0.28.0.upgrade.sql\n",
"psql -f herd.postgres.0.28.0-to-0.29.0.upgrade.sql\n",
"psql -f herd.postgres.0.29.0-to-0.30.0.upgrade.sql\n",
"psql -f herd.postgres.0.30.0-to-0.31.0.upgrade.sql\n",
"psql -f herd.postgres.0.31.0-to-0.32.0.upgrade.sql\n",
"psql -f herd.postgres.0.32.0-to-0.33.0.upgrade.sql\n",
"psql -f herd.postgres.0.33.0-to-0.34.0.upgrade.sql\n",
"psql -f herd.postgres.0.34.0-to-0.35.0.upgrade.sql\n",
"psql -f herd.postgres.0.35.0-to-0.36.0.upgrade.sql\n",
"psql -f herd.postgres.0.36.0-to-0.37.0.upgrade.sql\n",
"psql -f herd.postgres.0.37.0-to-0.38.0.upgrade.sql\n",
"psql -f herd.postgres.0.38.0-to-0.39.0.upgrade.sql\n",
"psql -f herd.postgres.0.39.0-to-0.40.0.upgrade.sql\n",
"psql -f herd.postgres.0.40.0-to-0.41.0.upgrade.sql\n",
"psql -f herd.postgres.0.41.0-to-0.42.0.upgrade.sql\n",
"psql -f herd.postgres.0.42.0-to-0.43.0.upgrade.sql\n",
"psql -f herd.postgres.0.43.0-to-0.44.0.upgrade.sql\n",
"psql -f herd.postgres.0.44.0-to-0.45.0.upgrade.sql\n",
"psql -f herd.postgres.0.45.0-to-0.46.0.upgrade.sql\n",
"psql -f herd.postgres.0.46.0-to-0.47.0.upgrade.sql\n",
"psql -f herd.postgres.0.47.0-to-0.48.0.upgrade.sql\n",
"psql -f herd.postgres.0.48.0-to-0.49.0.upgrade.sql\n",
"psql -f herd.postgres.0.49.0-to-0.50.0.upgrade.sql\n",
"psql -f herd.postgres.0.50.0-to-0.51.0.upgrade.sql\n",
"psql -f herd.postgres.0.51.0-to-0.52.0.upgrade.sql\n",
"psql -f herd.postgres.0.52.0-to-0.53.0.upgrade.sql\n",
"psql -f herd.postgres.0.53.0-to-0.54.0.upgrade.sql\n",
"psql -f herd.postgres.0.54.0-to-0.55.0.upgrade.sql\n",
"psql -f herd.postgres.0.55.0-to-0.56.0.upgrade.sql\n",
"psql -f herd.postgres.0.56.0-to-0.57.0.upgrade.sql\n",
"psql -f herd.postgres.0.57.0-to-0.58.0.upgrade.sql\n",
"psql -f herd.postgres.0.58.0-to-0.59.0.upgrade.sql\n",
"psql -f herd.postgres.0.59.0-to-0.60.0.upgrade.sql\n",
"psql -f herd.postgres.0.60.0-to-0.61.0.upgrade.sql\n",
"psql -f herd.postgres.0.61.0-to-0.62.0.upgrade.sql\n",
"psql -f herd.postgres.0.62.0-to-0.63.0.upgrade.sql\n",
"psql -f herd.postgres.0.63.0-to-0.64.0.upgrade.sql\n",
"psql -f herd.postgres.0.64.0-to-0.65.0.upgrade.sql\n",
"psql -f herd.postgres.0.65.0-to-0.66.0.upgrade.sql\n",
"psql -f herd.postgres.0.66.0-to-0.67.0.upgrade.sql\n",
"psql -f herd.postgres.0.67.0-to-0.68.0.upgrade.sql\n",
"psql -f herd.postgres.0.68.0-to-0.69.0.upgrade.sql\n",
"psql -f herd.postgres.0.69.0-to-0.70.0.upgrade.sql\n",
"psql -f herd.postgres.0.70.0-to-0.71.0.upgrade.sql\n",
"psql -f herd.postgres.0.71.0-to-0.72.0.upgrade.sql\n",
"psql -f herd.postgres.0.72.0-to-0.73.0.upgrade.sql\n",
"psql -f herd.postgres.0.73.0-to-0.74.0.upgrade.sql\n",
"psql -f herd.postgres.0.74.0-to-0.75.0.upgrade.sql\n",
"psql -f herd.postgres.0.75.0-to-0.76.0.upgrade.sql\n",
"psql -f herd.postgres.0.76.0-to-0.77.0.upgrade.sql\n",
"psql -f herd.postgres.0.77.0-to-0.78.0.upgrade.sql\n",
"psql -f herd.postgres.0.78.0-to-0.79.0.upgrade.sql\n",
"psql -f herd.postgres.0.79.0-to-0.80.0.upgrade.sql\n",
"psql -f herd.postgres.0.80.0-to-0.81.0.upgrade.sql\n",

Ultimately, the CloudFormation script still fails:
CREATE_FAILED | AWS::CloudFormation::WaitCondition | herdServerWaitCondition | WaitCondition timed out. Received 0 conditions when expecting 1
  | 14:52:36 UTC-0700 | CREATE_IN_PROGRESS | AWS::CloudFormation::WaitCondition | herdServerWaitCondition

How can I help restructure this go get around this error. Happy to contribute to the project, just need some idea how I could do so.
@nateiam
Copy link
Contributor

nateiam commented Nov 2, 2018 via email

@stevejhall stevejhall mentioned this issue Nov 27, 2018
Closed
@mrperson2015
Copy link

As a update, I ran through the demo install steps using herd-scripts-cloud-formation-0.88.0.jar. This issue is still very much open.

I am more interested in the data catalog piece than anything else. Our storage platform is not S3. This will catalog an event store in another system.

@ssaha01
Copy link

ssaha01 commented Jul 1, 2019

Nate,
Following your suggestion, I am trying to install herd-mdl through the provided installMDL.yml file. The installation is failing with the following error:

2019-07-01 12:11:23 UTC-0400 ice-poc-herd-mdl ROLLBACK_IN_PROGRESS The following resource(s) failed to create: [MdlStack]. . Rollback requested by user.
2019-07-01 12:11:23 UTC-0400 MdlStack CREATE_FAILED Embedded stack arn:aws:cloudformation:us-east-1:681636940503:stack/ice-poc-herd-mdl-MdlStack-16AJ6J361EUU6/ca1a8850-9c1a-11e9-8652-0eb49284c068 was not successfully created: The following resource(s) failed to create: [PrerequisitesPrimaryStack].

I tried to get the PowerUserAccess policy, but the link provided to get the sample policy is giving 404. Can you please help?

@nateiam
Copy link
Contributor

nateiam commented Jul 2, 2019

Hi @ssaha01 -

Glad you are looking at Herd! The best course of action here is to use the Herd-MDL install as discussed in the comment above. We also recommend setting 'DeployComponents' to 'Herd' -- unless you are interested in the integrated Hive Metastore and Presto cluster.

@rongwang is an engineer from our team and she can dig into the current resource creation issue.

@rongwang0930
Copy link
Contributor

@ssaha01 For better assist you, we'd like you retry the creation and send us more detail information.
1.can you retry your stack creation with Rollback on failure set to No (Rollback on failure setting is under advanced section when you create cft from console)?
2. find the actual failed nested stack inside PrerequisitesPrimaryStack, and send us a screenshot of the Events history for the failed nested stack

@dlutz2
Copy link

dlutz2 commented Feb 28, 2020 •
edited
Loading

I found that you can get below the EC2 user data limit by removing the parts of herd.localdb.template that loads the demo data: all the curl POSTs between lines 633-729.
That allows the CloudFormation script to run but herd-app doesn't start. Tomcat logs show a permission denied exception when trying to write "velocity.log". Caused by: java.io.FileNotFoundException: velocity.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_242]
at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_242]
at java.io.FileOutputStream.(FileOutputStream.java:213) ~[?:1.8.0_242]
at java.io.FileOutputStream.(FileOutputStream.java:133) ~[?:1.8.0_242]
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.FileAppender.(FileAppender.java:110) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.RollingFileAppender.(RollingFileAppender.java:79) ~[log4j-1.2.17.jar:?]
at org.apache.velocity.runtime.log.Log4JLogChute.initAppender(Log4JLogChute.java:118) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.Log4JLogChute.init(Log4JLogChute.java:85) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.LogManager.createLogChute(LogManager.java:157) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.LogManager.updateLog(LogManager.java:269) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeInstance.initializeLog(RuntimeInstance.java:871) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeInstance.init(RuntimeInstance.java:262) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeSingleton.init(RuntimeSingleton.java:112) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.app.Velocity.init(Velocity.java:74) ~[velocity-1.7.jar:1.7]
at org.finra.herd.service.helper.VelocityHelper.(VelocityHelper.java:41) ~[herd-service-0.116.0.jar:?]

Tomcat is apparently trying to write velocity.log in /root. Running Tomcat as root (very bad idea) works. Add to the cft script:
"/bin/sed -i 's/TOMCAT_USER=\"tomcat\"/TOMCAT_USER=\"root\"/g' /usr/share/tomcat8/conf/tomcat8.conf\n",

Also as far as I can tell, there is not a default user/password created for herd-ui, so it is not possible to login.

I am trying to switch to herd-mdl but our restricted environment doesn't allow creation of the required IAM entities, so having to rewrite the CloudFormation scripts to use existing entities. (which we did for the herd scripts but herd-mdl is more complex)

@TakumiHaruta
Copy link

TakumiHaruta commented Jan 19, 2021 •
edited
Loading

I tried to use version 0.136.0 and could pass through User data is limited to 16384 bytes error by replacing the db upgrade script with for loop.

          "psql -f herd.postgres.0.1.0.create.sql\n",
          "psql -f herd.postgres.0.1.0.refdata.sql\n",
          "psql -f herd.postgres.0.1.0.cnfgn.sql\n",
          "for i in {1..136}\n",
          "do\n",
          "  in=`expr $i + 1`\n",
          "  v_old=`printf %03d $i`\n",
          "  v_new=`printf %03d $in`\n",
          "  psql -f herd.postgres.0.${v_old}.0-to-0.${v_new}.0.upgrade.sql\n",
          "done\n",
          "psql -f activiti.postgres.create.engine.sql\n",
          "psql -f activiti.postgres.create.history.sql\n",
          "psql -f activiti.postgres.create.identity.sql\n",
          "psql -f quartz_tables_postgres.sql\n",
          "psql -f elasticsearch.configuration.values.sql\n",

But the UserData script still failed in the end. I got 404 during wget http://localhost:8080/herd-app/rest/buildInfo.
Sorry for the Japanese output, but it means you got 404.

/usr/bin/wget http://localhost:8080/herd-app/rest/buildInfo
--2021-01-19 01:48:49--  http://localhost:8080/herd-app/rest/buildInfo
localhost (localhost) をDNSに問いあわせています... 127.0.0.1
localhost (localhost)|127.0.0.1|:8080 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 404 
2021-01-19 01:48:49 エラー 404: (説明なし)。

I think herd.localdb.template doesn't work anymore. I just want to know what herd is like and don't want to use herd-mdi because it requires multiple resources, so I gave up on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@nateiam @dlutz2 @stevejhall @ssaha01 @rongwang0930 @mrperson2015 @TakumiHaruta