Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decimal vs double #3264

Open
meee1 opened this issue Jan 3, 2024 · 5 comments · May be fixed by #3275
Open

Decimal vs double #3264

meee1 opened this issue Jan 3, 2024 · 5 comments · May be fixed by #3275

Comments

@meee1
Copy link
Contributor

meee1 commented Jan 3, 2024

I've seen an issue where a cast from double to decimal causes an exception. Need to fix mavparam to remove all decimals

An int32 can be 100% represented by a double. So replace decimal with double. Only issue is with float rounding. So needs to be considered

Ie decimal.maxvalue < float.maxvalue

@meee1
Copy link
Contributor Author

meee1 commented Jan 3, 2024

@CraigElder did you end up getting a tlog for this one?

@meee1
Copy link
Contributor Author

meee1 commented Jan 3, 2024

image_2023_12_13T20_41_33_948Z-1.png

@meee1
Copy link
Contributor Author

meee1 commented Jan 3, 2024

@1702500308897-1.JPEG

@EosBandi
Copy link
Collaborator

EosBandi commented Jan 3, 2024

can we assume that if the value is larger that decimal.maxvalue then we can ignore conversion issues and cast it directly to double ?
my concerns : how can be a parameter value this small/large ? I suspects that flt_max is used somewhere... have to check if float.maxvalue to double then back to float will result the same number. if yes then it is easy, if not then it needs some more logic....

@EosBandi
Copy link
Collaborator

EosBandi commented Jan 4, 2024

float a = 	3.402823466e+38F;
float max_flt = Single.MaxValue;
double b = (double)a;
float c = (float)b;
Console.WriteLine("A :" + a);
Console.WriteLine("Max_flt :"+max_flt);
Console.WriteLine("B: "+b);
Console.WriteLine("C :"+c);
if (a == c) Console.WriteLine(" a == c");
if (a == max_flt) Console.WriteLine(" a == max_flt");
A :3.402823E+38
Max_flt :3.402823E+38
B: 3.40282346638529E+38
C :3.402823E+38
 a == c
 a == max_flt

The only remaining question is the first one: can we assume that if the value is larger that decimal.maxvalue then we can ignore conversion issues and cast it directly to double.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants