Added the ndcg metric [WIP]

[kamalojasv181]

Related #2631

Description: This is the implementation [WIP] for the NDCG metric.

Check list:

• Create a basic implementation.
• Handle rating ties.
• Write and parameterize tests.
• Add comments about its use
• Add exponential implementation
• Add Docs

[vfdev-5]

@kamalojasv181 thanks for the PR. I left few comments on few points, but I haven't yet explored the code to compute ndcg.

Can you please write few tests vs scikit-learn as ref ?

[vfdev-5]

Thanks a lot for the update @kamalojasv181 !
I left few other comments on the implementation and the API.

Let's also start working on docs and tests

[kamalojasv181]

Thanks for all the feedback. I will revert with a pull request addressing everything!

[vfdev-5]

Thanks for all the feedback. I will revert with a pull request addressing everything!

You can just continue working with this pull request, no need to revert anything.

[vfdev-5]

Thanks for the updates @kamalojasv181 !
I have few other code update suggestions.

[kamalojasv181]

@vfdev-5 there are a bunch of things I did in this commit:

1. Put the data on GPU for tests
2. Replaced torch.add with +.
3. Changed the naming and ordering as per ignite as (y_pred, y_true ....)
4. Added function to handle ties and wrote corresponding tests(A part of it is un-vectorized, TODO: Think of implementation to vectorise it.)
5. Added check for log_base > 0 and not equal to 1.
6. Added tests for log_base and exponential.
7. Separated the _ndcg_smaple_score and _dcg_smaple_score from the NDCG class.
   NOTE: log_base tests could have been done with the other tests, but I put them separately so that if sklearn changes their code, then we know about it (as the tests would be failing)

If there is anything else, lemme know before I can finally add some comments against the class and documentation.

[vfdev-5]

So, there is no way to make it vectorized == without for-loop ?

[kamalojasv181]

I havent checked yet. For now I have added this implementation. It's a TODO

[vfdev-5]

@kamalojasv181 thanks for the updates, I left few nits.
Next step is to write few tests for distributed configuration. @puhuk please provide few guidelines here on how to write them. Thanks !

[puhuk]

@vfdev-5 @kamalojasv181

Belows are checklist for test in ddp configuration

• Data generating with different random seed per each process
  • torch.random_seed(12 + rank + i)
• Generated data size per each rank should be n_iters * batch_size to update with its batch_size in update()
  • y_true = torch.randint(0, n_classes, size=(n_iters * batch_size,))
• Update data with batch style in update()
  • y_pred[i * batch_size : (i+1) * batch_size]
• Check generated data from each processes are gathered for computing (after running Engine)

  •    engine.run(data=data, max_epochs=n_epochs)
       y_true = idist.all_gather(y_true)
       y_preds = idist.all_gather(y_preds)

• Check calculated metric is same as reference
  • pytest.approx(calculated_value) == reference_value

The whole process should be seem like (Or you can refer from ignite/tests/ignite/metrics/test_accuracy.py)

ignite/tests/ignite/metrics/test_accuracy.py

Line 488 in 26f7cec

def _test_distrib_integration_list_of_tensors_or_numbers(device):

acc = Accuracy(is_multilabel=True, device=metric_device)

# data generation
torch.manual_seed(12 + rank)
y_true = torch.randint(0, n_classes, size=(n_iters * batch_size,)).to(device)
y_preds = torch.rand(n_iters * batch_size, n_classes).to(device)

# update each batch
def update(engine, i):
            return (
                y_preds[i * batch_size : (i + 1) * batch_size, :],
                y_true[i * batch_size : (i + 1) * batch_size],
            )

# Initialize Engine
engine = Engine(update)
acc = Accuracy(device=metric_device)
acc.attach(engine, "acc")

data = list(range(n_iters))
engine.run(data=data, max_epochs=n_epochs)

# all gather data
y_pred = idist.all_gather(y_pred)
y = idist.all_gather(y)

res = engine.state.metrics["acc"]

# calculate reference value with scikit learn and compare
true_res = sklearn.metrics.accuracy_score(y_true.cpu().numpy(), torch.argmax(y_preds, dim=1).cpu().numpy())
assert pytest.approx(res) == true_res

[vfdev-5]

@kamalojasv181 Why do you return tuple of 2 lists instead of a tuple of two tensors ?

[kamalojasv181]

I have taken inspiration from the code provided by @puhuk . Here each element of the list is a batch. We feed our engine one batch at a time; hence using a list is also ok. To maintain uniformity across the code, I have kept it this way.

[vfdev-5]

Oh, I see, he provided a wrong link. Yes, in accuracy we also a test on list of tensors and numbers but this is untypical. Here is a typical example

ignite/tests/ignite/metrics/test_accuracy.py

Lines 412 to 463 in 26f7cec

	def _test_distrib_integration_multilabel(device):
	rank = idist.get_rank()
	def _test(n_epochs, metric_device):
	metric_device = torch.device(metric_device)
	n_iters = 80
	batch_size = 16
	n_classes = 10
	torch.manual_seed(12 + rank)
	y_true = torch.randint(0, 2, size=(n_iters * batch_size, n_classes, 8, 10)).to(device)
	y_preds = torch.randint(0, 2, size=(n_iters * batch_size, n_classes, 8, 10)).to(device)
	def update(engine, i):
	return (
	y_preds[i * batch_size : (i + 1) * batch_size, ...],
	y_true[i * batch_size : (i + 1) * batch_size, ...],
	)
	engine = Engine(update)
	acc = Accuracy(is_multilabel=True, device=metric_device)
	acc.attach(engine, "acc")
	data = list(range(n_iters))
	engine.run(data=data, max_epochs=n_epochs)
	y_true = idist.all_gather(y_true)
	y_preds = idist.all_gather(y_preds)
	assert (
	acc._num_correct.device == metric_device
	), f"{type(acc._num_correct.device)}:{acc._num_correct.device} vs {type(metric_device)}:{metric_device}"
	assert "acc" in engine.state.metrics
	res = engine.state.metrics["acc"]
	if isinstance(res, torch.Tensor):
	res = res.cpu().numpy()
	true_res = accuracy_score(to_numpy_multilabel(y_true), to_numpy_multilabel(y_preds))
	assert pytest.approx(res) == true_res
	metric_devices = ["cpu"]
	if device.type != "xla":
	metric_devices.append(idist.device())
	for metric_device in metric_devices:
	for _ in range(2):
	_test(n_epochs=1, metric_device=metric_device)
	_test(n_epochs=2, metric_device=metric_device)

[vfdev-5]

@sadra-barikbin can you check why tests/ignite/metrics/test_precision.py::test_binary_input[None] is failing systematically ?

E       assert array([1.]) == approx([1.0 ± 1.0e-06, 0.0 ± 1.0e-12])
E         Impossible to compare arrays with different shapes.
E         Shapes: (2,) and (1,)

tests/ignite/metrics/test_precision.py:130: AssertionError

[puhuk]

@kamalojasv181 Hi, do you need any help to finalize this PR? Please feel free to let me and @vfdev-5 know :)

[ili0820]

Any updates on this ? If it is not finished yet, I'd love to contribute @vfdev-5

[vfdev-5]

Yes, this PR is not finished, unfortunately. @ili0820 if you can help with getting it landed it would be great!

[ili0820]

after reviewing all the comments,
As far as I understand, the testing part is the last and only problem that has been not resolved yet,
this seems resolved and
this is the problem.
Is this correct? @vfdev-5

[vfdev-5]

after reviewing all the comments, As far as I understand, the testing part is the last and only problem that has been not resolved yet, this seems resolved and this is the problem. Is this correct? @vfdev-5

@ili0820 yes, it remains few things here:

• https://github.com/pytorch/ignite/pull/2632/files#r930048810
• test for distributed configuration

In case if you are not familiar with DDP, please check: https://pytorch-ignite.ai/tutorials/advanced/01-collective-communication/. As for testing best practices, we would like to use now distributed fixture as here:

ignite/tests/ignite/metrics/test_ssim.py

Line 226 in f2b1183

def test_distrib_integration(distributed, metric_device):

if you have any questions, you can reach out to us on Discord in #start-contributing channel.