broken source file broke deployed package

[jbothma]

I somehow managed to have the source file modified in #58 end up being a json file instead of .csv as the URL indicates.

I think some time when os-data-importers ran, it then didn't collect useful data and overwrote the package I had deployed to OpenSpending

What would be a sensible way to avoid this? I guess the json was somehow interpreted as pretty bad CSV and assume none of the processors errored out so it thought it's a successful run and did the update.

Is there a way to add some demands like "these fields must be present and not blank" and/or "there must be at least X rows"?

[jbothma]

11:54 Brook Elgie @brew
mmm... I'm not sure there's anything you can add at your end to sanity check the input sources before they hit the pipelines. There is post-processing. Let me see where that's applied.
11:55 JD Bothma @jbothma
I guess I can add a custom processor - been meaning to do one for a unique key check
11:56 Brook Elgie @brew
Yeah, the 'post' processing step is actually quite early in the flow. It's part of the first flow for denomalizing the data. Mexico uses a custom processor, so you can use that as an example.