Limit the number of in-flight tasks

[kokil01]

I want to limit the number of in-flight tasks at a given point in time. Given a setup where I create a task for each line in a file (e.g. say each line is a url in the file)

for url in urls {
  task::spawn(async move {
    process_url(url).await;
  }
}

But this will create as many tasks as number of urls. Say I want to limit to only 100 urls being processed at a time. My first attempt is to use a semaphore

let sem = Arc::new(Semaphore::new(10));
for url in urls {
  let sem = Arc::clone(&sem);
  task::spawn(async move {
    let _permit = sem.acquire().await;
    process_url(url).await;
  }
}

This works at limiting the number of process_url in flight but still creates unbounded number of tasks. How can I use this semaphore to limit the number of tasks? I would have to acquire a SemaphorePermit before the spawn call but Drop it after the await of process_url but there is no way to pass anything in the async block of spawn function.

[hawkw]

Try

let sem = Arc::new(Semaphore::new(10));
for url in urls {
  let permit = Arc::clone(&sem).acquire_owned().await;
  task::spawn(async move {
    let _permit = permit;
    process_url(url).await;
  }
}

This way, the loop will wait until permits are available before spawning a new task, but the permits are not dropped until the spawned tasks complete.

[kokil01]

It worked! Thank you.

[hawkw]

Glad I could help! :)