json-iterator

Low level iterator on the records inside large JSON file.

Problem

How to read records from a very large and nested JSON file without loading the whole file in a memory?

If a JSON file is

• Very large
• The record to read is at deepest nested location
• May contain invalid json record

We cannot use json.Marshal() in this case, as it will load everything into memory causing out of memory error.

Solution

Register the type decoder with the iterator and call Next() till HasNext() returns false.

iterator := jsoniterator.NewJsonRecordsIterator("fruits.json")
iterator.RegisterGoTypeDecoder("{.trees.[.{.fruits", func(d *json.Decoder, m map[string]interface{}) (interface{}, error) {
    var fruit Fruit
    err := d.Decode(&fruit)
    // enrich fruit with data from m
    return fruit, err
})
for iterator.HasNext() {
    item := iterator.Next()
    if fruit, ok := item.(Fruit); ok {
        fmt.Println(fruit)
    }
}

See complete example