Fix nanflag support for min/max in mx-inlines.cc (bug #67714)

This changeset fixes a wrong result for min/max operations on dense matrices
with NaN values, which can be reproduced with:

[m,i] = max ([1,2,3;4,3,NaN;4,5,6], [], 2, "includenan")

* mx-inlines.cc: Rework the logic of nan-handling as described above.

* max.cc: Add BISTs

Author: pr0m1th3as

libinterp/corefcn/max.cc

@@ -1108,6 +1108,19 @@ is real or complex.  For elements with equal magnitude, a second comparison by
 %!assert (min ([1i 2 -3 4]), 1i)
 %!assert (min ([-2+i, 2-i]), 2-1i)
 
+## Test nanflag with dense arrays
+%!test
+%! [m,i] = min ([1,2,3;4,3,NaN;4,5,6], [], 2, "includenan");
+%! assert (m, [1; NaN; 4]);
+%! assert (i, [1; 3; 1]);
+%! [m,i] = min ([1,2,3;4,NaN,NaN;4,5,6], [], 2, "includenan");
+%! assert (m, [1; NaN; 4]);
+%! assert (i, [1; 2; 1]);
+%!test
+%! x = magic (3);
+%! x(2, 3) = NaN;
+%! assert (min (x, [], 2, "includenan"), [1; NaN; 2]);
+
 ## Test input validation
 %!error min ()
 %!error min (1, 2, 3, 4)
@@ -1403,6 +1416,19 @@ is real or complex.  For elements with equal magnitude, a second comparison by
 %!assert (min ([1i 2 -3 4]), 1i)
 %!assert (min ([-2+i, 2-i]), 2-1i)
 
+## Test nanflag with dense arrays
+%!test
+%! [m,i] = max ([1,2,3;4,3,NaN;4,5,6], [], 2, "includenan");
+%! assert (m, [3; NaN; 6]);
+%! assert (i, [3; 3; 3]);
+%! [m,i] = max ([1,2,3;4,NaN,NaN;4,5,6], [], 2, "includenan");
+%! assert (m, [3; NaN; 6]);
+%! assert (i, [3; 2; 3]);
+%!test
+%! x = magic (3);
+%! x(2, 3) = NaN;
+%! assert (max (x, [], 2, "includenan"), [8; NaN; 9]);
+
 ## Test input validation
 %!error max ()
 %!error max (1, 2, 3, 4)