Minimum matches

[rderidder-lda]

Hello,
Question regarding the minimum number of matches..
Since I have a training file, which contains distinct and match records listed out...
I was assuming that:

1. I should see (as a definite minimum) every listed training match, as matches in the resulting entity map
2. I should never see a match in the entity map, that occurs in the distinct training list

is this not a correct assumption?
I just ran the process, with 2k match records in training, and 6k distincts.. but the process came back with BlockingError('No records have been blocked together. Is the data you are trying to match like the data you trained on? If so, try adding more training data.')

Thanks for any explanation / tips!
R

[rderidder-lda]

As a follow up.. I removed about 5.6k of the distincts to see if that would help, and it still resulted in the above error.
I then removed nearly all the records from training, and did get the process to run without the above error.. I checked the resulting entity map for all 'match' entries and verify no 'distinct' entries.. here are the results:
total number of records in the collection being matched: 93594
number of records in the entity map: 66289
number of distinct canon_ids in the entity map: 20548

match PAIRS in training data: 128
same match PAIR found in entity map (thought I could expect ALL): 91
(note that they might not be direct matches, but both match records are at least matching to the same third record)

distinct PAIRS in training data: 220
distincts PAIR found in entity map (thought I could expect NONE): 22
(note that they might not be direct matches, but both distincts are at least matching to the same third record)

Why doesn't dedupe ensure the training data is reflected accurately in the resulting entity map? I mean, training data should be correct right?

And, why does it error with no blocking, saying to add more training data, when I put a LOT of training data in it? I had to reduce the training matches, to get it to block records? confuses me - so any help appreciated!

[rderidder-lda]

for the second question.. i've been playing with the data, and I may have found that it is a data issue with one of the records possibly causing the whole thing to not work when its included in the training data.. Still narrowing down. Will update on that one.
So for now, maybe just help me out on the first question around why the training data isn't reflected in the results 100%