JSMN_TOKNEXT

[matejkraus]

When traversing JSON array or object, it is good to know where the given structure ends.

For this reason, I added the compiler directive JSMN_TOKNEXT, which adds information about the next token.

For clarity, I added a tokendump example:

make tokendump
./tokendump library.json
index    type[size] start-end toknext value
-------------------------------------------
  0:    OBJECT[008]:    0- 367:  22
  1:    STRING[001]:    5-   9:   0: name
  2:    STRING[000]:   13-  17:   0: jsmn
  3:    STRING[001]:   23-  31:   0: keywords
  4:    STRING[000]:   35-  39:   0: json
  5:    STRING[001]:   45-  56:   0: description
  6:    STRING[000]:   60- 171:   0: Minimalistic JSON parser/tokenizer in C. It can be easily integrated into resource-limited or embedded projects
  7:    STRING[001]:  177- 187:   0: repository
  8:    OBJECT[002]:  192- 264:  13
  9:    STRING[001]:  199- 203:   0: type
 10:    STRING[000]:  207- 210:   0: git
 11:    STRING[001]:  218- 221:   0: url
 12:    STRING[000]:  225- 259:   0: https://github.com/zserge/jsmn.git
 13:    STRING[001]:  269- 279:   0: frameworks
 14:    STRING[000]:  283- 284:   0: *
 15:    STRING[001]:  290- 299:   0: platforms
 16:    STRING[000]:  303- 304:   0: *
 17:    STRING[001]:  310- 318:   0: examples
 18:     ARRAY[001]:  321- 344:  20
 19:    STRING[000]:  328- 339:   0: example/*.c
 20:    STRING[001]:  349- 356:   0: exclude
 21:    STRING[000]:  360- 364:   0: test

[matejkraus]

tokendump example for union { end, toknext }:

index    type[size] start-end toknext value
-------------------------------------------
  0:    OBJECT[008]:    0-  22:  22
  1:    STRING[001]:    5-   9:   9: name
  2:    STRING[000]:   13-  17:  17: jsmn
  3:    STRING[001]:   23-  31:  31: keywords
  4:    STRING[000]:   35-  39:  39: json
  5:    STRING[001]:   45-  56:  56: description
  6:    STRING[000]:   60- 171: 171: Minimalistic JSON parser/tokenizer in C. It can be easily integrated into resource-limited or embedded projects
  7:    STRING[001]:  177- 187: 187: repository
  8:    OBJECT[002]:  192-  13:  13
  9:    STRING[001]:  199- 203: 203: type
 10:    STRING[000]:  207- 210: 210: git
 11:    STRING[001]:  218- 221: 221: url
 12:    STRING[000]:  225- 259: 259: https://github.com/zserge/jsmn.git
 13:    STRING[001]:  269- 279: 279: frameworks
 14:    STRING[000]:  283- 284: 284: *
 15:    STRING[001]:  290- 299: 299: platforms
 16:    STRING[000]:  303- 304: 304: *
 17:    STRING[001]:  310- 318: 318: examples
 18:     ARRAY[001]:  321-  20:  20
 19:    STRING[000]:  328- 339: 339: example/*.c
 20:    STRING[001]:  349- 356: 356: exclude
 21:    STRING[000]:  360- 364: 364: test

[pt300]

So either a new value in a token struct or end/toknext changing meaning depending on the type. From those two options first one makes more sense. The union will make traversal of the token array unnecessarily complicated.
But, what could be even better is a function that would calculate the token size for you. If toknext is hidden behind an ifdef then that function would just return that value, otherwise it would manually traverse the array to find the next token and return its index. You get to choose between more memory or more processing time, but the API would stay 'stable'.