Performance Issues Due to SDK

[pnadolny13]

This issue should be closed by meltano/sdk#1779 and meltano/sdk#1962 in the SDK so its mostly a reminder to bump the version once those fixes are released.

I was using this target and noticed a few performance issues similar to those I'd seen in target-snowflake:

• I noticed a ton of sink re-initialization logs
• preparing tables seemed to take a long time. For me it was taking ~30s per table. This causing pain because the inserts would eventually fail (tap issues) but I had to wait so long to find out on each iteration.

cc @visch @edgarrmondragon

[edgarrmondragon]

For me it was taking ~30s per table.

Wow that's a lot 😬. Thanks for diving in @pnadolny13!

[visch]

Eeek nice issue @pnadolny13 . 30s per table is high, how many columns is it? I'd guess over 50-100? If it's the number of columns the cache issue would fix it but I think just optimizing our prepare table function would do it too, I think there's a meta data call per column (I'm guessing as I haven't dove in but I know the reserved column name test takes a while to run and it has a ton of columns)

[edgarrmondragon]

I published version 0.32.0b2 of the SDK if you guys wanna give it a try

[pnadolny13]

@edgarrmondragon it looks like theres a release out with part of this but meltano/sdk#1779 is still pending. I actually think that one is whats causing the 30s per table lag since its constantly running the same metadata query. Would you mind reviewing that?

[pnadolny13]

Closed by meltano/sdk#1779 and #208