Start / Stop Queued Tracking from CLI

[asyslinux]

Hello I have a problem with high loaded matomo.
I divide tasks by time(archive/optimization/queue processing).

When I start archive process or optimization i need to temporary disable moving queue from redis to mysql from cli, not from web interface.

I do:

1. update plugin_setting set setting_value='0' where setting_name='processDuringTrackingRequest';
2. Restart php-fpm

And sometimes workers stop moving queue from redis to mysql, sometimes not. Sometimes need wait 2 or 3 hours when queue is stopped.

Ok, but when i want to start queues processing I do:

1. update plugin_setting set setting_value='1' where setting_name='processDuringTrackingRequest';
2. Restart php-fpm

And queue processing during request does not start workers automatically.

I look:

root@anl:/var/www/anl# su www-data -c "/var/www/anl/console queuedtracking:monitor"
Queue is enabled
Request sets in the queue will be processed automatically after a tracking request
Up to 16 workers will be used
Processor will start once there are at least 25 request sets in the queue
4470266 (288425+273982+277090+285114+283500+274067+273271+284790+283130+278360+281069+281945+284532+277483+271149+272359) request sets left in queue. 17.14G used
4471065 (288482+274034+277146+285156+283553+274122+273330+284850+283191+278401+281109+281983+284568+277539+271197+272404) request sets left in queue. 17.15G used
memory (41.58G peak). 0 workers active.

0 Workers active, but I change in MySQL setting already for start 16 workers processing.

And I can not use /usr/bin/php /var/www/anl/console queuedtracking:process this command by cron, because 1 worker give around 100-150 rps of queue processing, but new visits come to redis with 500-900 rps speed. In my case, I can use only 16 workers processing with enabled processDuringTrackingRequest. But have trouble with disable end enable from command line and directly in mysql table. This start/stop need for maintaince archive/optimizations processess with database.

If I not divide this processes, i every day have troubles with mysql locking(All is optimized including innodb_adaptive_hash_index). I have around 1 000 000 000 actions per months on website :). But in real life we save only 2 last weeks range - for us enough. Hardware is good - 4xXeon 80vCPU / 192GB RAM / SSD.

When maintance task is finished, next task is start queue processing from redis to mysql.

Now I am every day same cron go to web interface and push Save button for activate queue processing...

May be someone can help with stop and start queue processing from command line, not from web interface?

And I have question about 16 workers limit, i have some vCPUs and 32 innodb write threads. I try to change limit to 32 workers in plugins/QueuedTracking/SystemSettings.php file and in mysql plugin setting , but when workers is started I see only 16 workers in cli monitor. Other 16 workers have 0+0+0+0+0+0 values in cli monitor....

I mean that it would be nice to be able to put the data from queues(redis) in mysql into a pause and then take it off.

Thank You.

[tsteur]

Maybe on top call ./console cache:clear and ./console queuedtracking:lock-status --unlock=$queueId this might help.

Otherwise a command to stop/start can be useful 👍 I doubt we would work on this soon though as we don't really have a use case for it yet (and we usually recommend to use CLI processing)

[asyslinux]

I have one more question. I know Actions core plugin save Downloads/Outlinks. We have so much downloads actions. I look in documentation, but I not find how to disable Downloads actions.

For example we track all actions only for domain example.com / www.example.com , and on this web site we have so much links - cdn.example.com/pics/.jpg , cdn.example.com/pics/.png , images.example.com/* , and some external downloads actions. This is useless information for us(calculation of download actions), but in our database we have around 12 000 000 download actions per 24 hour. We want decrease load on piwik by disable save and track download actions. If developers of matomo can improve functional and make option for disable tracking Downloads actions - this is can be useful for high load web site with much image/video content.

For example - we want use matomo with commercial plugins same heatmap, but now we can not stabilize standard functionality on our traffic, and if we buy heatmap plugin for example, I think this is create one more load on matomo. So now we are thinking about optimizing and disabling the not required functionality.

Thank You.