Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Selecting parameter from Response Correlation Viewer causing util abort #3168

Closed
oysteoh opened this issue Mar 30, 2022 · 2 comments
Closed
Labels

Comments

@oysteoh
Copy link
Contributor

oysteoh commented Mar 30, 2022

@oysteoh commented on Tue Mar 29 2022

Clicking around selectin various parameters i hit this one

DEBUG:urllib3.connectionpool:http://127.0.0.1:51838 "GET /ensembles/febf080e-fcd3-4292-bc2c-cd885fa98b6b/records/RMSGLOBPARAMS:KVKH_CSAND_18 HTTP/1.1" 200 4752
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:51838

Error message: buffer_fread: read beyond the length of the buffer (212 exceeds 170)

See file: /tmp/ert_abort_dump.ooh.20220329-132758.log for more details of the crash.

[ooh@st-linrgsn313 model]$ cat /tmp/ert_abort_dump.ooh.20220329-132758.log 


Abort called from: buffer_fread (/tmp/jenkins-komodo-f_komodo/workspace/x-build-some-other/bleeding-py36-rhel7/komodo-releases/cache/ecl-main/lib/util/buffer.cpp:186)

Error message: buffer_fread: read beyond the length of the buffer (212 exceeds 170)


--------------------------------------------------------------------------------
Stack trace (most recent call last):
#31   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74026dc372, in _PyGen_Send
#30   Object "/lib64/libpython3.6m.so.1.0", at 0x7f740276a817, in _PyEval_EvalFrameDefault
#29   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74026dc372, in _PyGen_Send
#28   Object "/lib64/libpython3.6m.so.1.0", at 0x7f740276a817, in _PyEval_EvalFrameDefault
#27   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74026dc372, in _PyGen_Send
#26   Object "/lib64/libpython3.6m.so.1.0", at 0x7f740276a817, in _PyEval_EvalFrameDefault
#25   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74026dc372, in _PyGen_Send
#24   Object "/lib64/libpython3.6m.so.1.0", at 0x7f740276a817, in _PyEval_EvalFrameDefault
#23   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74026dc372, in _PyGen_Send
#22   Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402769146, in _PyEval_EvalFrameDefault
#21   Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402774b52, in PyEval_EvalFrameEx
#20   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74027747e9, in PyEval_EvalFrameEx
#19   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74027739a3, in PyEval_EvalFrameEx
#18   Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402769146, in _PyEval_EvalFrameDefault
#17   Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402774b52, in PyEval_EvalFrameEx
#16   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74027747e9, in PyEval_EvalFrameEx
#15   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74027739a3, in PyEval_EvalFrameEx
#14   Object "/lib64/libpython3.6m.so.1.0", at 0x7f740276a2fc, in _PyEval_EvalFrameDefault
#13   Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402774b52, in PyEval_EvalFrameEx
#12   Object "/lib64/libpython3.6m.so.1.0", at 0x7f74027747e9, in PyEval_EvalFrameEx
#11   Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402773e0e, in PyEval_EvalFrameEx
#10   Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402769146, in _PyEval_EvalFrameDefault
#9    Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402774a2e, in PyEval_EvalFrameEx
#8    Object "/lib64/libpython3.6m.so.1.0", at 0x7f7402708346, in _PyCFunction_FastCallDict
#7    Object "/prog/res/komodo/bleeding-py36-rhel7/root/lib64/python3.6/site-packages/res/_lib.cpython-36m-x86_64-linux-gnu.so", at 0x7f73f09916b7, in PyInit__lib
#6    Object "/prog/res/komodo/bleeding-py36-rhel7/root/lib64/python3.6/site-packages/res/_lib.cpython-36m-x86_64-linux-gnu.so", at 0x7f73f09878d1, in conf_instance_get_path_error(conf_instance_struct const*)
#5    Object "/prog/res/komodo/bleeding-py36-rhel7/root/lib64/python3.6/site-packages/res/_lib.cpython-36m-x86_64-linux-gnu.so", at 0x7f73f0900da9, in enkf_plot_gen_kw_load
#4    Object "/prog/res/komodo/bleeding-py36-rhel7/root/lib64/python3.6/site-packages/res/_lib.cpython-36m-x86_64-linux-gnu.so", at 0x7f73f0900c01, in enkf_plot_gen_kw_vector_load(enkf_plot_gen_kw_vector_struct*, enkf_fs_struct*, bool, int)
#3    Object "/prog/res/komodo/bleeding-py36-rhel7/root/lib64/python3.6/site-packages/res/_lib.cpython-36m-x86_64-linux-gnu.so", at 0x7f73f090cf11, in enkf_node_try_load
#2    Object "/prog/res/komodo/bleeding-py36-rhel7/root/lib64/python3.6/site-packages/res/_lib.cpython-36m-x86_64-linux-gnu.so", at 0x7f73f090c40d, in enkf_fs_fread_node(enkf_fs_struct*, buffer_struct*, char const*, enkf_var_type, int, int)
#1    Object "/prog/res/komodo/bleeding-py36-rhel7/root/lib64/python3.6/site-packages/ecl/.libs/libecl.so", at 0x7f73f0d91c8b, in buffer_fread
#0    Object "/prog/res/komodo/bleeding-py36-rhel7/root/lib64/python3.6/site-packages/ecl/.libs/libecl.so", at 0x7f73f0d80cbe, in util_abort__


@oysteoh commented on Tue Mar 29 2022

Quite common to get selecting different parameters in the test-case actually....


@oysteoh commented on Tue Mar 29 2022

Actually consistent across a group of parameters called TRAJSHIFT:xxxxxxx. Loading data from those causing the error mentioned. Selecting the default ensemble empty charts is rendered, selecting any ensembles except default seems to trigger this exception.

@BjarneHerland
Copy link
Contributor

Which testcase?

@oysteoh
Copy link
Contributor Author

oysteoh commented Mar 30, 2022

We was presented with a case to test towards webviz and they reported errors. The case should be working

This is caused by .lock - files being present in the storage due to ERT crashing and leaving storage in a way it is not possible to use.

A likely reason to this case failing is that the storage is in an unsynchronised state (the "fingers crossed" warning is given when trying to access plot data to a case. Lock files are observed in the storage folder, although there should not be any open ert instances).

enkf_fs_fread_node calls driver->load_node with a preallocated buffer (size 100) which will be reallocated given information in storage by following:

void ert::block_fs_driver::load_node(const char *node_key, int report_step,
int iens, buffer_type *buffer) {
char *key = block_fs_driver_alloc_node_key(node_key, report_step, iens);
bfs_type *bfs = this->get_fs(iens);
block_fs_fread_realloc_buffer(bfs->block_fs, key, buffer);
free(key);
}

The realloc_buffer tries to read data from storage, and this could be the reason that it fails, as it can not open the necessary files.

void block_fs_fread_realloc_buffer(block_fs_type *block_fs,
const char *filename, buffer_type *buffer) {
block_fs_aquire_rlock(block_fs);
{
file_node_type *node =
(file_node_type *)hash_get(block_fs->index, filename);
buffer_clear(buffer); /* Setting: content_size = 0; pos = 0; */
pthread_mutex_lock(&block_fs->io_lock);
block_fs_fseek_node_data(block_fs, node);
buffer_stream_fread(buffer, node->data_size, block_fs->data_stream);
//file_node_verify_end_tag( node , block_fs->data_stream );
pthread_mutex_unlock(&block_fs->io_lock);
buffer_rewind(buffer); /* Setting: pos = 0; */
}
block_fs_release_rwlock(block_fs);
}

Note that there were multiple cases in the ert setup, and only for the cases that had lock files present this would fail.

@oysteoh oysteoh closed this as completed Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants