Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

riak kv 3.2.0 OTP25 turns off when stanchion starts #1139

Open
Konstantin74R opened this issue Jul 25, 2023 · 2 comments
Open

riak kv 3.2.0 OTP25 turns off when stanchion starts #1139

Konstantin74R opened this issue Jul 25, 2023 · 2 comments

Comments

@Konstantin74R
Copy link

When i first start riak, then riak-cs and stanchion, riak kv self turns off. (riak ping -- Node [email protected] is not responding to pings).
Version riak 3.2.0 OTP25
riak-cs-3.0.1.OTP22
stanchion-3.0.0.OTP22

How can I fixed? After reboot OS it normal works, but before restarting riak applications.

In logs
2023-07-25T12:01:17.107424+00:00 [warning] <0.786.0>@riak_kv_vnode:log_key_amnesia/7:4493 Inbound clock entry for <<35,9,254,249,50,191,134,141>> in <<"moss.users">>/<<"SE4FRXVTWHVXCBEFAQW2">> greater than local.Epochs: {In:210001 Local:0}. Counters: {In:1 Local:0}. 2023-07-25T12:01:17.109281+00:00 [warning] <0.786.0>@riak_kv_vnode:log_key_amnesia/7:4493 Inbound clock entry for <<35,9,254,249,50,191,134,141>> in <<"moss.users">>/<<"SE4FRXVTWHVXCBEFAQW2">> greater than local.Epochs: {In:210001 Local:0}. Counters: {In:1 Local:0}. 2023-07-25T12:01:17.110775+00:00 [warning] <0.786.0>@riak_kv_vnode:log_key_amnesia/7:4493 Inbound clock entry for <<35,9,254,249,50,191,134,141>> in <<"moss.users">>/<<"SE4FRXVTWHVXCBEFAQW2">> greater than local.Epochs: {In:210001 Local:0}. Counters: {In:1 Local:0}. 2023-07-25T12:01:17.112909+00:00 [info] <0.671.0>@riak_repl_app:stop/1:139 Stopped application riak_repl 2023-07-25T12:01:17.114183+00:00 [info] <0.404.0>@riak_kv_app:prep_stop/1:273 Stopping application riak_kv - marked service down. 2023-07-25T12:01:17.114289+00:00 [info] <0.404.0>@riak_kv_app:prep_stop/1:277 Unregistered pb services 2023-07-25T12:01:17.115158+00:00 [info] <0.404.0>@riak_kv_app:prep_stop/1:282 unregistered webmachine routes 2023-07-25T12:01:17.115246+00:00 [info] <0.404.0>@riak_kv_app:prep_stop/1:284 all active put FSMs completed 2023-07-25T12:01:17.115463+00:00 [info] <0.404.0>@riak_kv_app:stop/1:295 Stopped application riak_kv. 2023-07-25T12:01:17.143085+00:00 [info] <0.840.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {205523667749658222872393179600727299639115513856,3} for full rebuild on next restart 2023-07-25T12:01:17.143167+00:00 [info] <0.844.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {182687704666362864775460604089535377456991567872,3} for full rebuild on next restart 2023-07-25T12:01:17.143220+00:00 [info] <0.1463.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1050454301831586472458898473514828420377701515264,3} for full rebuild on next restart 2023-07-25T12:01:17.143265+00:00 [info] <0.841.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {228359630832953580969325755111919221821239459840,3} for full rebuild on next restart 2023-07-25T12:01:17.143322+00:00 [info] <0.1465.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1073290264914881830555831049026020342559825461248,3} for full rebuild on next restart 2023-07-25T12:01:17.145574+00:00 [info] <0.1451.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1027618338748291114361965898003636498195577569280,3} for full rebuild on next restart 2023-07-25T12:01:17.151856+00:00 [info] <0.840.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {228359630832953580969325755111919221821239459840,3} for full rebuild on next restart 2023-07-25T12:01:17.151913+00:00 [info] <0.844.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {205523667749658222872393179600727299639115513856,3} for full rebuild on next restart 2023-07-25T12:01:17.151959+00:00 [info] <0.1463.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1073290264914881830555831049026020342559825461248,3} for full rebuild on next restart 2023-07-25T12:01:17.152623+00:00 [info] <0.841.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {251195593916248939066258330623111144003363405824,3} for full rebuild on next restart 2023-07-25T12:01:17.152679+00:00 [info] <0.1465.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1096126227998177188652763624537212264741949407232,3} for full rebuild on next restart 2023-07-25T12:01:17.167194+00:00 [info] <0.1451.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1050454301831586472458898473514828420377701515264,3} for full rebuild on next restart 2023-07-25T12:01:17.171056+00:00 [info] <0.840.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {251195593916248939066258330623111144003363405824,3} for full rebuild on next restart 2023-07-25T12:01:17.171115+00:00 [info] <0.844.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {228359630832953580969325755111919221821239459840,3} for full rebuild on next restart 2023-07-25T12:01:17.171164+00:00 [info] <0.1463.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1096126227998177188652763624537212264741949407232,3} for full rebuild on next restart 2023-07-25T12:01:17.171219+00:00 [info] <0.841.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {274031556999544297163190906134303066185487351808,3} for full rebuild on next restart 2023-07-25T12:01:17.171802+00:00 [info] <0.1465.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1118962191081472546749696200048404186924073353216,3} for full rebuild on next restart 2023-07-25T12:01:17.181151+00:00 [info] <0.1451.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1073290264914881830555831049026020342559825461248,3} for full rebuild on next restart 2023-07-25T12:01:17.240497+00:00 [info] <0.215.0>@riak_core_app:stop/1:118 Stopped application riak_core.

In log stanchion
2023-07-25 11:53:47.276 [warning] <0.838.0>@stanchion_app:check_admin_creds:77 admin.secret is ignored. 2023-07-25 11:59:37.510 [warning] <0.838.0>@stanchion_app:check_admin_creds:77 admin.secret is ignored. 2023-07-25 12:01:47.939 [error] <0.838.0>@stanchion_app:start:59 Couldn't connect to Riak: {riak_connect_failed,{{tcp,econnrefused},"10.32.101.175",8087}} 2023-07-25 12:01:47.939 [error] <0.839.0> CRASH REPORT Process <0.839.0> with 0 neighbours exited with reason: {tcp,econnrefused} in gen_server:init_it/6 line 358 2023-07-25 12:01:47.939 [error] <0.837.0> CRASH REPORT Process <0.837.0> with 0 neighbours exited with reason: {{riak_connect_failed,{{tcp,econnrefused},"10.32.101.175",8087}},{stanchion_app,start,[normal,[]]}} in application_master:init/4 line 138 2023-07-25 12:01:47.939 [info] <0.681.0> Application stanchion exited with reason: {{riak_connect_failed,{{tcp,econnrefused},"10.32.101.175",8087}},{stanchion_app,start,[normal,[]]}}

@Bob-The-Marauder
Copy link
Contributor

When running Stanchion and KV on the same node, I've sometimes noticed permissions issues with the /tmp/erl_pipes folder - both KV and Stanchion like to claim it as their own which means that the other cannot write to it. I'm not sure if this will fix your problem but maybe try moving changing permissions on that folder so that anyone can write to it?

Another option is to look at Riak CS 3.1.0 or 3.2.1 where Stanchion is integrated into all Riak CS nodes, so if one instance fails, another one takes over.

@Konstantin74R
Copy link
Author

Thank you, I will try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants