fail to load searchd response / slice bounds out of range

[ghost]

Hi guys,

Hope you are all well !

I have 2 major issues with the go-sdk and managed to reproduce them into the following repository.
Ref. https://github.com/x0rzkov/manticore-gosdk-issue

In a nutshell, the two issues are:

1. On some queries, I have a slice bounds out of range error.
   To reproduce, please see the README.md of the repository above.

runtime error: slice bounds out of range [:774778400] with capacity 22412    
/usr/local/go/src/runtime/panic.go:106 (0x432eb2)    
	goPanicSliceAcap: panic(boundsError{x: int64(x), signed: true, y: y, code: boundsSliceAcap})    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/sphinxql.go:311 (0x10a139c)    
	(*apibuf).getMysqlStrLen: result := string((*buf)[:lng])    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/sphinxql.go:234 (0x10a0a5a)    
	(*Sqlresult).parserow: strValue := buf.getMysqlStrLen()    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/sphinxql.go:74 (0x109f901)    
	(*Sqlresult).parseChain: if !rs.parserow(source) {    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/sphinxql.go:20 (0x10a62b5)    
	parseSphinxqlAnswer.func1: if rs.parseChain(answer) {    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/client.go:225 (0x1093649)    
	(*Client).netQuery: return parser(&answer), nil    
/home/ubuntu/lucmichalski/paper2code/pkg/manticore/manticore.go:398 (0x109503f)    
	(*Client).Sphinxql: blob, err := cl.netQuery(commandSphinxql,    
/home/ubuntu/lucmichalski/paper2code/main.go:3639 (0x118b2a7)    
	controllersSearch: res2, err2 := cl.Sphinxql(query)    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/context.go:161 (0xc9b61a)    
	(*Context).Next: c.handlers[c.index](c)    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/recovery.go:83 (0xcaedaf)    
	RecoveryWithWriter.func1: c.Next()    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/context.go:161 (0xc9b61a)    
	(*Context).Next: c.handlers[c.index](c)    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/logger.go:241 (0xcadee0)    
	LoggerWithConfig.func1: c.Next()    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/context.go:161 (0xc9b61a)    
	(*Context).Next: c.handlers[c.index](c)    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/gin.go:409 (0xca53f5)    
	(*Engine).handleHTTPRequest: c.Next()    
/home/ubuntu/go/pkg/mod/github.com/gin-gonic/gin@v1.6.3/gin.go:367 (0xca4b0c)    
	(*Engine).ServeHTTP: engine.handleHTTPRequest(c)    
/home/ubuntu/go/pkg/mod/github.com/qor/session@v0.0.0-20170907035918-8206b0adab70/gorilla/gorilla.go:125 (0xb2b591)    
	Gorilla.Middleware.func1: handler.ServeHTTP(w, req.WithContext(ctx))    
/usr/local/go/src/net/http/server.go:2012 (0x76eb73)    
	HandlerFunc.ServeHTTP: f(w, r)    
/usr/local/go/src/net/http/server.go:2807 (0x772002)    
	serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)    
/usr/local/go/src/net/http/server.go:1895 (0x76d97b)    
	(*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)    
/usr/local/go/src/runtime/asm_amd64.s:1373 (0x465670)    
	goexit: BYTE	``````x90	// NOP    
%CODE_BLOCK%`    
    
2. Sphinxql: failed to read searchd response    
When I dockerize manticore and make a query to manticore from the local env; everything works, and when I do exactly the same thing/query/task/action within dockerized application it triggers a failed to read searchd response.    
    
All the details to reproduce are also available in the README of the repository.    
    
Hope that you will find quickly the issues related to these bugs ^^    
    
Cheers,    
X   
   
 
 

[klirichek]

As I can guess by line numbers, that is code published some revisions ago.
In the near past it was fix about line length, and actual tip of master now is 7d072dc
Could you check, if your issue is still actual agains this head?

[ghost]

I just checked my code and the go.mod file, and it is on the right revision https://github.com/x0rzkov/manticore-gosdk-issue/blob/master/go.mod#L11

aka 7d072dc

I mean it is still actual

[klirichek]

ok, will check.
(I asked because according to your backtrace - line sphinxql.go:311 has no 'result := string((*buf)[:lng])' in 7d072dc, it was so in 0e8dbff. And that is quite important, since 7d072dc contains fix of old well-known crash in mysql processing, and that is why I asking about. (in 7d072dc that would be sphinxql.go:320)

[ghost]

Thanks, my whole website is blocked because of these 2 issues, so yeah if you can sort it out it gonna be awesome :-) ^^

[ghost]

Any updates mate ?

[klirichek]

Not yet, need to complete current backlog of C++ tasks first.

[githubmanticore]

fix error in mysql len parsing

That closes #15 via commit 70a49dc

[ghost]

Hi guys,

Thanks for the fix, I am testing it right now and will come back with a feedback during the day.

But remains the second bug, quiet annoying for my rss feeds based on manticoresearch:

When I dockerize manticore and make a query to manticore from the local env; everything works, 
and when I do exactly the same thing/query/task/action within dockerized application it 
triggers a failed to read searchd response.

I gave you all the details to reproduce here.
https://github.com/x0rzkov/manticore-gosdk-issue#second-bug-sphinxql-failed-to-read-searchd-response

Cheers,
X

[klirichek]

Sounds interesting to look, but right now I am not able to do it fast
(previous one I've reproduced on bare metal, running your repro in IDE on macbook where I have no docker engine. Will check later on linux with docker).
However just wonder: if you run queries as sphinxql commands - why don't you just use mysql proto directly?
"database/sql" works well to connect to daemon and run your queries directly, without need to proxy them via sphinxql api command.
And that is also looking as good way to check if the problem is really specific to api proto implementation (if connecting to the dockerized instance using database/sql on mysql port works, but api not), or to some internal network implementation details of the daemon (if connecting to real mysql works, but connect to manticore mysql port not), or some more generic missconfiguration (if neither works in the docker at all)

[ghost]

Hi,

I tried with the mysql driver it works but not for my search query as I am using multiple facets as arguments. I do not get facets information with the mysql driver.

Please check that issue as I have no clue how to debug that more than the repo I gave you.

Tell me that you found the issue as my whole website is stuck because of this last bug. ^^

Thanks in advance.

Cheers,
X

[sanikolaev]

@x0rzkov

I tried with the mysql driver it works but not for my search query as I am using multiple facets as arguments. I do not get facets information with the mysql driver.

https://github.com/go-sql-driver/mysql doesn't support multiple results, but https://stackoverflow.com/a/28577876/591867 works for me:

snikolaev@dev:~$ cat test_luc.go
package main

import (
    "flag"
    "fmt"

    "github.com/ziutek/mymysql/autorc"
    "github.com/ziutek/mymysql/mysql"
    _ "github.com/ziutek/mymysql/thrsafe"
)

type ScanFun func(int, []mysql.Row, mysql.Result) error

func RunSQL(hostport, user, pass, db, cmd string, scan ScanFun) error {
    conn := autorc.New("tcp", "", hostport, user, pass, db)

    err := conn.Reconnect()
    if err != nil {
        return err
    }

    res, err := conn.Raw.Start(cmd)
    if err != nil {
        return err
    }

    rows, err := res.GetRows()
    if err != nil {
        return err
    }

    RScount := 0
    scanErr := error(nil)

    for {
        if scanErr == nil {
            func() {
                defer func() {
                    if x := recover(); x != nil {
                        scanErr = fmt.Errorf("%v", x)
                    }
                }()
                scanErr = scan(RScount, rows, res)
            }()
        }

        if res.MoreResults() {
            res, err = res.NextResult()
            if err != nil {
                return err
            }
            rows, err = res.GetRows()
            if err != nil {
                return err
            }
        } else {
            break
        }

        RScount++
    }
    return scanErr
}

func main() {
    host := flag.String("host", "localhost:9306", "define the host where the db is")
    user := flag.String("user", "root", "define the user to connect as")
    pass := flag.String("pass", "", "define the pass to use")
    db := flag.String("db", "information_schema", "what db to default to")

    sql := flag.String("sql", "select * from t where match('abc') facet a", "Query to run")

    flag.Parse()

    scan := func(rcount int, rows []mysql.Row, res mysql.Result) error {
        if res.StatusOnly() {
            return nil
        }

        for idx, row := range rows {
            fmt.Print(rcount, "-", idx, ") ")
            for i, _ := range row {
                fmt.Print(row.Str(i))
                fmt.Print(" ")
            }
            fmt.Println("")
        }
        return nil
    }

    fmt.Println("Host - ", *host)
    fmt.Println("Db   - ", *db)
    fmt.Println("User - ", *user)

    if err := RunSQL(*host, *user, *pass, *db, *sql, scan); err != nil {
        fmt.Println(err)
    }
}
snikolaev@dev:~$ go get github.com/ziutek/mymysql/autorc
snikolaev@dev:~$ go run test_luc.go
Host -  localhost:9306
Db   -  information_schema
User -  root
0-0) 2810903486202382289 0 1
0-1) 2810903486202382290 1 0
0-2) 2810903486202382291 1 2
1-0) 0 1
1-1) 1 2

[ghost]

Hi guys,

Updated my code for using your snippets, and it works from the docker container:
https://paper2code.com/feed?f_subjects=5,4

Sounds like docker is not the cause. Isn't it ?

Cheers,
X

[klirichek]

Sounds like docker is not the cause. Isn't it ?

Well, if you please check one more test:
place the daemon near your application (into same container), and add distributed index with only one remote agent - original daemon which should work from the docker.
I.e. from schema:

[you app -> go-sdk] -> [daemon_backend]

it will became:

[your app -> go-sdk -> daemon_proxy] -> [daemon_backend]

This way we may determine the thing whether problem is in go-sdk code, or on daemon side (since in second variant proxy daemon will connect to production using the same binary protocol as implemented in go-sdk). If it works, you may even use such schema as temporary solution (proxy daemon doesn't need to have any local indexes)

[ghost]

Can you do that with the repository I gave you ? I think it is gonna be better and faster if you do that task on your side.

[sanikolaev]

It might be better, but not sure it will be faster (in terms of the issue resolution) as we (and Alexey in particular) are extremely busy with other tasks, so any help from the community is highly appreciated.

[ghost]

Sounds like docker is not the cause. Isn't it ?

Well, if you please check one more test:
place the daemon near your application (into same container), and add distributed index with only one remote agent - original daemon which should work from the docker.
I.e. from schema:

[you app -> go-sdk] -> [daemon_backend]

it will became:

[your app -> go-sdk -> daemon_proxy] -> [daemon_backend]

This way we may determine the thing whether problem is in go-sdk code, or on daemon side (since in second variant proxy daemon will connect to production using the same binary protocol as implemented in go-sdk). If it works, you may even use such schema as temporary solution (proxy daemon doesn't need to have any local indexes)

Can you be more clear about that ? I do not get how I can get the paper2code server and manticoresearch daemon running together in a docker container.

Sorry but I do not understand your request

[klirichek]

The thing which is curious to check - whether go-sdk can't connect to the daemon because of the problem in go-sdk, or because of the problem in the daemon.

To check this most obvious way is to use not go-sdk, but another daemon as API client (this very binary protocol implemented in this go-sdk is the one used by daemon to connect to agents).

Assuming that port forwarding is set up correct (i.e. when you connect to port YYY of node XXX it will route it to legal local process waiting for proper interface and proper port inside the docker, which is trivial when starting dockerized app, but I just clearify it to be explicit), you can use any API client, including copy of the daemon (since it also speaks with same protocol). You need searchd executable in the container (whole 'manticore' package and environment is not necessary, only daemon binary is enough), and short config which has one distributed index containing one agent line which is exactly location of your dockerized real container, like:

searchd
{
  pid_file = searchd.pid
  binlog_path =
}
index your_index
{
  type = distributed
  agent = real.host.address
}

configure your app to connect to 'localhost:9312'.
Add a line to execute ./searchd -c path/to/config before or right after (to not stole pid=1) of your application in the same startup script.

Daemon will start in classical daemon way (via double-fork and detach from the session) and will search locally (inside container) port 9306 and 9312 (first for mysql, second for the rest). Querying index 'your_index' on such address will translate any queries to api binary protocol and send to the host defined as agent. In case of queries already in api binary it will works just as a kind of proxy in this simplest case (since it is only one agent, no mirrors, no custom aggretations, just resend query to onther host). For your application in this case - it will connect to localhost:9312 as the host. And proxy will actually connect to remote real.host.address:9312.

So, if problem in go-sdk, then such proxying should work well.
Worse if this configuration also fails - that means that daemon couldn't work with daemonized agents, and if so, that is much more serious problem we have to fix (not on go-sdk side in this case).

[ghost]

@sanikolaev

The snippet is not working for this query, it is missing all facets

@klirichek I ll try to install a local manticore and test the go version/dockerized version today.

[sanikolaev]

Aren't these facets?

[ghost]

Mea culpa, still trying to figure out how to recode the website ^^

[tomatolog]

seems you use multiple queries with facet at each of them. That will not work at daemon. You could use only one query with multiple facets at that query or multiple queries without facet .

[ghost]

Hi guys,

Hope you are all well !

I managed to convert my code for mysql queries, and it works fine.
So I do not know if it is better to wait for the new version of sdk for further development.

Cheers,
X

[sanikolaev]

Great, if just pure mysql works for you in Go just use it :)
I'm wondering if you had time to check the new experimental Go client based on the HTTP protocol we discussed in Slack.

[ghost]

I ll check it this week ! Keep you updated

[Ekliptor]

I also got the error failed to read searchd response (status=0, ver=256, len=88136, read=42920)

Simply limiting the results or adding a SelectClause for attributes solved the problem for me.