Heroku pg:copy can get caught in an error state, infinitely printing "▸ not found"

[msakrejda]

From @Taytay (see heroku/cli#545 )

---

We regularly create and copy Hobby DBs as part of our deploy process. Occasionally, one of these processes will get hung on a pg:copy command. Here's the output from the one I just killed:

00:01:29.050 Creating heroku-postgresql:standard-0 on <OUR_APP>... $50/month
00:01:29.050 The database should be available in 3-5 minutes.
00:01:29.050 ! CAUTION: The database will be empty. If upgrading, you can transfer
00:01:29.050 !          data from another database with pg:copy.
00:01:29.050 Use `heroku pg:wait` to track status.
00:01:29.051 Waiting for postgresql-convex-40295...
00:03:23.043 Creating postgresql-convex-40295... done
00:03:23.044 Created postgresql-convex-40295 as DATABASE_URL
00:03:23.630 Attaching HEROKU_POSTGRESQL_CYAN from <OUR_APP> to <OUR_APP> as HEROKU_POSTGRESQL_CYAN... done
00:03:24.657 Status: available
Status: available
00:03:24.657 Transferring <OLD_APP>:DATABASE to <OUR_APP>:DATABASE...
00:03:43.815 Progress: 0MB/692MB
Progress: 5MB/692MB
Progress: 14MB/692MB
00:03:43.816  ▸    not found
00:03:49.260 
00:03:49.260  ▸    not found
00:03:55.019 
00:03:55.019  ▸    not found
00:04:00.792 
00:04:00.792  ▸    not found
00:04:06.565 
00:04:06.565  ▸    not found
00:04:12.296 
00:04:12.297  ▸    not found
00:04:18.152 
00:04:18.152  ▸    not found
00:04:23.998 
00:04:23.999  ▸    not found
00:04:29.787 
00:04:29.787  ▸    not found
00:04:35.618 
00:04:35.618  ▸    not found
00:04:41.433 
00:04:41.433  ▸    not found
00:04:47.225 
00:04:47.225  ▸    not found
00:04:53.029 
00:04:53.030  ▸    not found
00:04:58.804 
00:04:58.804  ▸    not found
00:05:04.629 
00:05:04.630  ▸    not found

It goes on like that for another 16 hours, so I think it's safe to say it's dead. :)
The copy command looks like this:

heroku pg:copy DATABASE $NEW_DB_ATTACHMENT_NAME --app $HEROKU_APP_NAME --confirm $HEROKU_APP_NAME

In short, we're not doing anything odd. I think that the new DB is disappearing out from under the copy command somehow, and after that, it doesn't know how to recover.

[msakrejda]

@Taytay do you know what version of the CLI and the plugin you were using? I've looked at this and the wait should give up after 20 errors. We should probably still handle 404s specially and exit early (e.g., if the relevant database is destroyed mid-copy), but I don't see how we would go into an indefinite retry loop with the current code.