Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove JSON Dependency #1812

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Remove JSON Dependency #1812

wants to merge 6 commits into from

Conversation

barronalex
Copy link
Collaborator

A basic JSON parser that allows us to remove the nlohmann/json dependency.

Should be pretty much complete with the exception of unicode support.

mlx/io/load.h Outdated Show resolved Hide resolved
mlx/io/load.h Show resolved Hide resolved
mlx/io/safetensors.cpp Outdated Show resolved Hide resolved
Copy link
Member

@awni awni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks for the update! Left a couple minor comments.. but otherwise g2g!

Copy link
Member

@angeloskath angeloskath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very nice addition. Very well done.

I left a few comments on things I missed the first time around. They mostly have to do with edge case error checking for invalid JSON so not absolutely critical but it 'd be nice to fix before merging.

json parse_json(char* s, int length) {
membuf sbuf(s, length);
std::istream stream(&sbuf);
std::string os(s, length);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgotten.

return result;
}

struct membuf : std::streambuf {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

s >> std::ws >> ch;
if (ch == '}') {
break;
} else if (ch != ',') {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is something that can be fixed later but unfortunately this is a tiny bit out of spec. It is also one of the most annoying parts of the json spec so I don't really mind if we don't fix it immediately but we should at least put a TODO unfortunately.

The following is unfortunately not valid JSON ["a", "b",] (notice the extra comma) but it would be parsed without error here. Both for object and array.

Comment on lines +123 to +127
std::string str = "";
for (int i = 0; i < 3; i++) {
str += s.get();
}
if (str != "ull") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
std::string str = "";
for (int i = 0; i < 3; i++) {
str += s.get();
}
if (str != "ull") {
char buff[3] = {0};
if (s.read(buff, 3); !std::equal(buff, buff+3, "ull")) {

This is a total nitpick but the string may do a heap allocation there (may even do more than 1 given the +=) and it really is just to check 3 characters so I think it is a bit of a an overkill. You could even do if (s.get() != 'u' || s.get() != 'l' || s.get() != 'l') {...

Comment on lines +24 to +28
if (num.find_first_of(".eE") != std::string::npos) {
return json(std::stod(num));
} else {
return json(std::stol(num));
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (num.find_first_of(".eE") != std::string::npos) {
return json(std::stod(num));
} else {
return json(std::stol(num));
}
size_t pos;
json ret;
if (num.find_first_of(".eE") != std::string::npos) {
ret = json(std::stod(num, &pos));
} else {
ret = json(std::stol(num, &pos));
}
if (pos != num.size()) {
throw std::invalid_argument(...);
}
return std::move(ret);

The above is a bit of a problem because read_digits reads any character in [0-9-eE.] which means it could be not a number eg 1.2.3.4 and in this case we should fail. std::stod and friends return the number of characters processed so it is a fairly easy change to check for correctness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants