Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent Attributor::InvalidDefinition when running multi-threaded app under rainbows #174

Open
justingaylor opened this issue Aug 23, 2016 · 1 comment
Open

Intermittent Attributor::InvalidDefinition when running multi-threaded app under rainbows #174

justingaylor opened this issue Aug 23, 2016 · 1 comment

Comments

@justingaylor
Copy link
Contributor

When running under rainbows in multi-threaded mode (50 workers, 2 threads/worker) to investigate another issue, I've encountered the following intermittent errors.

Type: Attributor::InvalidDefinition, Message: Structure definition for type #<Class:0x00000004241fb8> is invalid. The following exception has occurred: #<RuntimeError: can't add a new key into hash during iteration>, Backtrace: 
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/attributor-5.0.2/lib/attributor/types/hash.rb:84:in `attributes'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/attributor-5.0.2/lib/attributor/types/model.rb:180:in `method_missing'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/attributor-5.0.2/lib/attributor/types/model.rb:139:in `block in validate'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/attributor-5.0.2/lib/attributor/types/model.rb:136:in `each'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/attributor-5.0.2/lib/attributor/types/model.rb:136:in `validate'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/attributor-5.0.2/lib/attributor/types/hash.rb:430:in `validate'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/attributor-5.0.2/lib/attributor/attribute.rb:215:in `validate'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request.rb:151:in `validate_params'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request_stages/validate_params_and_headers.rb:48:in `execute'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request_stages/request_stage.rb:83:in `execute_with_around'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request_stages/request_stage.rb:61:in `run'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request_stages/request_stage.rb:111:in `block in execute'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request_stages/request_stage.rb:110:in `each'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request_stages/request_stage.rb:110:in `execute'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request_stages/request_stage.rb:83:in `execute_with_around'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/request_stages/request_stage.rb:61:in `run'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/dispatcher.rb:87:in `block (2 levels) in dispatch'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/dispatcher.rb:86:in `each'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/dispatcher.rb:86:in `block in dispatch'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/notifications.rb:164:in `block in instrument'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/notifications.rb:164:in `instrument'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/notifications.rb:24:in `instrument'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/dispatcher.rb:81:in `dispatch'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/bootloader_stages/routing.rb:19:in `call'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/router.rb:17:in `call'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/router.rb:35:in `invoke'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/router/simple.rb:129:in `block (2 levels) in call'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/router/simple.rb:127:in `catch'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/router/simple.rb:127:in `block in call'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/router/simple.rb:126:in `each'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/router/simple.rb:126:in `call'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/router.rb:77:in `call'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/application.rb:115:in `block in call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/notifications.rb:164:in `block in instrument'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.7/lib/active_support/notifications.rb:164:in `instrument'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/notifications.rb:24:in `instrument'
  <APP_ROOT>/vendor/cache/praxis-ca6f964fc155/lib/praxis/application.rb:114:in `call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/stalin-0.2.0/lib/stalin/adapter/rack.rb:30:in `call'
  <APP_ROOT>/vendor/cache/global_session-66c8bdcf73b8/lib/global_session/rack.rb:146:in `call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rack-contrib-1.4.0/lib/rack/contrib/cookies.rb:44:in `call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/right_support-2.11.3/lib/right_support/rack/request_logger.rb:129:in `call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/right_support-2.11.3/lib/right_support/rack/request_tracker.rb:78:in `call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rack_injection-0.2.0/lib/rack_injection.rb:12:in `call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rack-contrib-1.4.0/lib/rack/contrib/runtime.rb:18:in `call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rainbows-4.7.0/lib/rainbows/max_body.rb:66:in `block in call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rainbows-4.7.0/lib/rainbows/max_body.rb:59:in `catch'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rainbows-4.7.0/lib/rainbows/max_body.rb:59:in `call'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rainbows-4.7.0/lib/rainbows/process_client.rb:44:in `process_loop'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rainbows-4.7.0/lib/rainbows/thread_pool.rb:41:in `sync_worker'
  <APP_ROOT>/vendor/bundle/ruby/2.1.0/gems/rainbows-4.7.0/lib/rainbows/thread_pool.rb:25:in `block (2 levels) in worker_loop'

My test sends 100 concurrent requests to the service, all the same exact request format. Of the 100 requests, 2 or 3 return "400 Bad Request" with the above validation error, while the rest of the requests return 200 OK.

Here's the response body for the failures:

{
  "name": "ValidationError",
  "summary": "Error loading params.",
  "errors": [
    "Error loading attribute $.params of type  from value {\"limit\"=>\"1\"}\nStructure definition for type #<Class:0x00000004241fb8> is invalid. The following exception has occurred: #<RuntimeError: can't add a new key into hash during iteration>"
  ],
  "cause": {
    "name": "Attributor::InvalidDefinition",
    "message": "Structure definition for type #<Class:0x00000004241fb8> is invalid. The following exception has occurred: #<RuntimeError: can't add a new key into hash during iteration>"
  }
}

It feels like this could be a problem with thread safety in the params hash processing in praxis/attributor, so I thought I would report here.

Please let me know if I should report this under praxis or here and how I can help. Cheers.
@justingaylor
Copy link
Contributor Author

I looked a bit into the code and found this is where the error is raised:
https://github.com/rightscale/attributor/blob/fbaae0a4fccd8ab04761ac1784eeae55eb7c01dc/lib/attributor/types/hash.rb#L83

And it appears @errors is only set here:
https://github.com/rightscale/attributor/blob/fbaae0a4fccd8ab04761ac1784eeae55eb7c01dc/lib/attributor/types/hash.rb#L116

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@justingaylor