Replace throttle implementation by external library

[jbigman]

No description provided.

[jbigman]

Don't mind about the third commit description, i missed a copy paste. it contains eslint adjustments.

[facundoolano]

why did you need to add this explicit throttle setup here? doesn't the setup in the internal request module apply here as well by just passing the throttle option?

[jbigman]

I thougth the same, but test fails if i don't edit processpage when it batch request fulldetails

[jbigman]

I wrote this implementation

const req = (resolve, reject) => requestLib(opts)
    .then((response) => resolve(response.body))
    .catch((error) => reject(error));

  let promiseToResolve = (resolve, reject) => req(resolve, reject);

  if (throttleParams !== undefined) {
    const throttle = pThrottle({
      limit: throttleParams.limit ? throttleParams.limit : 1000,
      interval: throttleParams.interval ? throttleParams.interval : 0
    });
    promiseToResolve = (resolve, reject) => throttle(req(resolve, reject));
  }
  return new Promise(promiseToResolve);

But we could use the throttling method everytime with default values set to {limit: 1000, interval : 0}

  const throttle = pThrottle({
    limit: throttleParams && throttleParams.limit ? throttleParams.limit : 1000,
    interval: throttleParams && throttleParams.interval ? throttleParams.interval : 0
  });
  return new Promise((resolve, reject) => throttle(requestLib(opts)
    .then((response) => resolve(response.body))
    .catch((error) => reject(error))));

[jbigman]

Both the limit and interval options must be specified in p-throttle usage.

That's why i force interval or limit with default values if one is missing

[facundoolano]

so a couple of things:

1. by default throttling should be off. we don't want to silently introduce a limit unless the client explicitly tell us to do so
2. it's ok to set a default to the missing argument if the other is specified.
3. see the related documentation on this repo. You should make sure we continue to support the same behavior, and update accordingly if there are new options or things to be considered (but it's best not to introduce a change in behavior or interface if not necessary). In particular, I think it's fine to continue to assume the interval is always seconds, and just take an integer as the throttle value, which will be assumed to be the request limit by second.
4. I need to review more closely that duplicated setting you have for the fullDetail, I want to understand why it's necessary

[jbigman]

in fact, p-throttle doesnt queue requests, i'll try to use throttledQueue instead

[facundoolano]

closing due to inactivity, but feel free to send a new one.