Multiprocessing per queue is not supported

[adulau]

Multiprocessing per queue is not supported. If you have to process a huge dataset and only a single process is handling a queue, you are time bound with DNS resolution or alike. It would be nice to ensure that you can handle multiple processes for handling a single queue.

[Rafiot]

FYI, it is supported on AIL by this module: https://github.com/CIRCL/AIL-framework/blob/master/bin/Helper.py

I will see if it is possible to implement it.

[aaronkaplan]

On May 28, 2015, at 3:52 PM, Alexandre Dulaunoy notifications@github.com wrote:

Multiprocessing per queue is not supported. If you have to process a huge dataset and only a single process is handling a queue, you are time bound with DNS resolution or alike. It would be nice to ensure that you can handle multiple processes for handling a single queue.

—
Reply to this email directly or view it on GitHub.

So concerning DNS in particular, we had a discussion on this already.
Tomas tried to implement threading here, but it turned out to be a bit awkward and ugly (a.k.a. complex).

So, we decided against this and reverted to processes and/or asynchronous DNS.
In fact, I'd like to see something like adns-tools for DNS lookups.

However, right now (at least for us) speed was not the main concern.

Any experience with adns in python?

[Rafiot]

Protip: never use threading in python. Been here, done that, bad idea.

My approach on AIL is to run processes poping values from a set and pushing the output on the next one.

I will dig into the code of intelmq to see if I can use the same library as in AIL.

[SYNchroACK]

The new version that is under testing (https://github.com/certtools/intelmq-beta) is possible to enable 'load_balance' which is in the reality a splitte.

https://github.com/certtools/intelmq-beta/blob/master/intelmq/lib/pipeline.py#L82

What do you think about the idea? Is not multi-thread, is "muti-process".

[aaronkaplan]

On May 28, 2015, at 4:35 PM, Raphaël Vinot notifications@github.com wrote:

Protip: never use threading in python. Been here, done that, bad idea.

ACK :)

My approach on AIL is to run processes poping values from a set and pushing the output on the next one.

I will dig into the code of intelmq to see if I can use the same library as in AIL.

okay. Cool! Thx a lot.

—
Reply to this email directly or view it on GitHub.

[SYNchroACK]

Parsers will have the load_balance option equal True, so they will send one event to cymru-1 and second one to cymru-2, and so on...

PS: manager graph will need the have a hierarchy structured to be more understandable.

Visualization of idea:

[aaronkaplan]

I strongly dislike that we create loops in the graph. Even if it is just loops in the underlying undirected graph (in your example, there are no cycles in the DAG - but the underlying undirected one has cycles).

[adulau]

The best is to simplify such approach. In AIL for example, multiple processes can read from the same queue or publisher. That's it. So depending of the resource of the machine you can start one or more processes per queue/publisher.

[SYNchroACK]

@aaronkaplan check the code and propose. Its a little bit hard to get a simple solution like the current one.

[aaronkaplan]

On May 28, 2015, at 5:17 PM, Alexandre Dulaunoy notifications@github.com wrote:

The best is to simplify such approach. In AIL for example, multiple processes can read from the same queue or publisher. That's it. So depending of the resource of the machine you can start one or more processes per queue/publisher.

Agreed.
That makes sense.

—
Reply to this email directly or view it on GitHub.

[Rafiot]

Here is my model for one queue in AIL

You can have multiple Queues subscribing to the same publisher.

[aaronkaplan]

+1 pub / sub seems a good way . Are pushes and pops atomic in redis?

[adulau]

Yes.

[aaronkaplan]

Let me maybe elaborate why I think the solution by @Rafiot seems better for me: because you can't make loops. Loops can bite us a lot if we are not careful to mark all the data which has already been processed. So what I do not yet understand in @Rafiot 's picture: if the data goes into the Set , is the pop atomic as well even for a whole message/event (seems like yes, because a list is a certain type of set (note: I am not a Redis guru))? Secondly, have we thought about using more of the pub/sub model of Redis in pushing and popping ? Is that what's actually implied in the picture? Third, how would you re-arrange the base bot class so that it follows this principle? What features does an abstract bot class need?

--> maybe some of these questions are answered by looking at the AIL source :)