Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: mathematical precision #938

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

cptbtptpbcptdtptp
Copy link
Collaborator

@cptbtptpbcptdtptp cptbtptpbcptdtptp commented Aug 10, 2022

The number in JS uses the IEEE754 standard: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number

IEEE754 conversion rules: https://www.h-schmidt.net/FloatConverter/IEEE754.html

$2^{53}\rightarrow 10^{15.95}$

@GuoLei1990 GuoLei1990 added bug Something isn't working math Math library labels Aug 10, 2022
@GuoLei1990 GuoLei1990 added this to the 0.9 milestone Aug 10, 2022
@GuoLei1990
Copy link
Member

GuoLei1990 commented Aug 10, 2022

Encoding is: $2^{\left( 2-v0\right) }\times \left( 1+v1\right)$
Maybe is $2^{52}-1\rightarrow 10^{15.65}$
Math.floor(15.653559774527022)=15
?

@GuoLei1990
Copy link
Member

Should fix Math unit test

out._x = _x * len;
out._y = _y * len;
out._z = _z * len;
out._onValueChanged && out._onValueChanged();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use out.set(_x * len, _y * len, _z * len); is better?

@GuoLei1990 GuoLei1990 modified the milestones: 0.9, 1.0 Nov 14, 2022
@GuoLei1990 GuoLei1990 added the medium priority Medium priority issue label Nov 14, 2022
@GuoLei1990 GuoLei1990 modified the milestones: 1.0, 1.1 May 18, 2023
@GuoLei1990 GuoLei1990 modified the milestones: 1.1, 1.2 Oct 7, 2023
@cptbtptpbcptdtptp cptbtptpbcptdtptp modified the milestones: 1.2, Backlog Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working math Math library medium priority Medium priority issue
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

2 participants