Attempting to cluster correlation outputs using OPTICS

[OmnipotentEntity]

Currently text-only attempt of clustering correlation outputs along lines. However, attempts are failing to find clusters in both the (newly written) ownership_corr and futurepos_corr outputs.

Also added a genmoves command to quickly plop stones on the board so that we can reach a point in the middle of the game more quickly, generates 10 moves at a time by default.

Solicitating feedback on the approach, especially for those of you who are more familiar with clustering algorithms than I am. (I'm just using the defaults in OPTICS, but perhaps a different xi parameter would be better? or a different algorithm entirely?)

[OmnipotentEntity]

Some examples:

On-going ko, note that the engine believes whoever wins the ko will lose territory on the other side of the board (presumably from ignoring a ko threat):

Area of uncertain ownership is correlated strongly to the area around itself, but anti-correlated to the ownership of the top group. Whomever got gote and more points up top doesn't get this area on the bottom.

Futurepos of the same spot, it's a little bit of a mess and difficult to interpret.

A little later in the same game, ownership map is a little uncertain in two large areas. However,

It shows a very strong anti-correlation between these two areas.

[OmnipotentEntity]

As requested in the Discord:

1. What does the correlation structure look like for the empty board, around various points in the corner/sides/center - how far does it radiate?

Regular ownership map:

Top left corner (Q16, marked):

Top center (K16, marked):

Top right 3-3 point (R17, marked):

Remark: it is very odd that the engine is correlating all four corners in this way and anti-correlating it with the sides and center. I would have expected two corners to be correlated with each other, and the opposite to be anti-correlated, mostly because cross openings seem to be slightly more rare.

It is interesting that the engine seems to believe that it is slightly less likely that the 4-4 and 3-3 point will be owned by the same person.

This final image also has at least some interesting anti correlation. The person who gets a top 3-3 probably isn't going to get the bottom side.

2. If two corners are occupied, and in a way that the policy is mostly indifferent between the remaining two, there should be a clear negative correlation between the remaining two?

This is what I thought as well. However it doesn't seem like the engine agrees?

Ownership:

Ownership Correlation with Q4, marked:

Gif:

I can't really explain why, I'm going to simply guess that the ownership correlation doesn't quite have a handle on situations like this yet.

3. After a 3-3 invasion, in a position where the policy is indifferent between blocking each way, there should be some interesting correlations between the ownership of the nearby two sides and between them and the corner?

Again, no. Which is clearly wrong, because even the ownership seems to have an opinion.

Ownership:

Correlation against R4 (marked):

[OmnipotentEntity]

So it seems that I have made a small error, which implied that the network was much more certain about ownership correlation than it actually is. This error was using cosine similarity vs inner product, which should be the same if the inputs are normalized, but the inputs are not normalized. The content isn't changed, just the degree of certainty. Here is the opening for example:

[shindavid]

How difficult would it be to create the empty board correlation map directly from self play ownership data, without the network?

[shindavid]

I know that some of the self-play games are handicap games. Are these games included for correlation training? If so, might they impact empty board maps?