Check all feeds before aborting

[bmos]

Queue until end and abort if any error was raised

[bmos]

going to add Brett's suggestions
https://discussion.dsausa.org/t/dsa-feed-not-updating-github-actions/31234/15?u=wil.t

[brettchalupa]

Nice! This really helps see how many are failing and what needs to be cleaned up.

[brettchalupa]

Great improvements! Looking good! This will make it easy to fix the busted feeds

[bmos]

I believe this to be done other than some connection and SSL errors.

[brettchalupa]

@bmos for the SSL errors, I think just commenting out those sites from the ini is the best option. I checked out a few and they have expired SSL certs, which IMO means we shouldn't include them.

[bmos]

Certainly, but we probably also want the action log to continue checking the other sites rather than getting 'stuck' like it did for redirects prior to this PR

[brettchalupa]

Working really well for me locally! I don't have merge power yet, but I think this will make it a lot easier to maintain into the future.

[bmos]

Any experience with async or parallel operations in Ruby?
Since it's talking to a bunch of different servers, the run time could easily be reduced.
This might actually be a better solution to #31

[brettchalupa]

@bmos yes! Breaking the array of feeds up into threads could help. Here's a generic example:

# Sample array
arr = [1, 2, 3, 4, 5]

# Define the operation to perform on each element
def process_element(element)
  sleep 1 # Simulate IO-bound operation
  element * 2
end

# Create threads and collect results
results = []
threads = arr.map do |element|
  Thread.new do
    results << process_element(element)
  end
end

# Wait for all threads to finish
threads.each(&:join)

# Print the results
puts results # => [2, 4, 6, 8, 10] (order might vary)

It should, in theory, be more efficient. It might require some slicing up the arrays into multiple. Let me know if I can help adapt given the example above is maybe too generic. 😂

[bmos]

Wow that was so much easier than implementing threading in any other language I have tried.

Can confirm, it's WAY faster with multiple workers. although, rather predictably, if you use more than 1 worker the console outputs get all scrambled (which defeats the purpose).

So it'll need more work to compile and print each line as a unit rather than doing each checkmark in realtime.

That being said, I'd still consider this ready to merge since it'll make a big difference right away to the usability of this validation script and improvements can be make later to enable the benefits of threading.

[bmos]

It now runs all the way through the whole file without getting stuck.