Skip to content
This repository has been archived by the owner on Jan 29, 2024. It is now read-only.

Low throughput #11

Open
EdvardD opened this issue Jul 20, 2020 · 0 comments
Open

Low throughput #11

EdvardD opened this issue Jul 20, 2020 · 0 comments

Comments

@EdvardD
Copy link

EdvardD commented Jul 20, 2020

I'm trying to use the library to process messages in the subscription with big number of messages. I have this code:

#include <chrono>
#include <memory>
#include <list>
#include <spdlog/spdlog.h>
#include <spdlog/sinks/stdout_sinks.h>
#include "pubsuber/Pubsuber.h"
#include "base/logging.h"

using namespace std::chrono;

int main() {
        pubsuber::ClientOptions opts = pubsuber::ClientOptions::CreateDefault("my-team");

        auto sink = std::make_shared<spdlog::sinks::stdout_sink_mt>();
        sink->set_pattern("%D-%T.%f [%n] [%t] [%L]: %v");
        opts.SetLogSink(sink);
        opts.SetLogLevel(spdlog::level::info);

        pubsuber::ClientPtr client;

        pubsuber::RetryCountPolicy countPolicy{4};
        pubsuber::MaxRetryTimePolicy timePolicy{30s};
        pubsuber::ExponentialBackoffPolicy backoffPolicy{250ms, 3s, 2.0};

        client = pubsuber::Client::Create(std::move(opts), std::move(timePolicy), std::move(countPolicy), std::move(backoffPolicy));
        client->SetLogLevel(spdlog::level::info);

        auto subscription = client->GetSubscription("MY-CHANNEL");


        std::list<pubsuber::MessagePtr> messages;
        std::mutex mmm;
        size_t count = 0;
        subscription->Receive([&messages, &count, &mmm](pubsuber::MessagePtr&& m) {
                std::unique_lock l(mmm);
                if (++count % 1000 == 0) {
                        LOG(Info, "count: " << count << " message_size: " << m->Payload().size());
                        messages.clear();
                }
                messages.push_back(std::move(m));
                messages.back()->Ack();
        });

        while (true) {}

  return 0;
}

When I start the application there is a long queue of unacked messages (~100K) in the subscription.
I see that throughput doesn't allow to read them fast: I see throughput 1K messages in a minute.
Is this an expected speed to process messages? Is it possible to setup library to get a higher throughput?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant