javascript: Acceptance tests do not include tokens

[mpkorstanje]

🤔 What's the problem you've observed?

The acceptance tests for Javascript currently do verify if the parser produces the right tokens:

gherkin/javascript/Makefile

Lines 13 to 16 in 1cc2ef0

	ASTS = $(patsubst ../testdata/%,acceptance/testdata/%.ast.ndjson,$(GOOD_FEATURE_FILES))
	PICKLES = $(patsubst ../testdata/%,acceptance/testdata/%.pickles.ndjson,$(GOOD_FEATURE_FILES))
	SOURCES = $(patsubst ../testdata/%,acceptance/testdata/%.source.ndjson,$(GOOD_FEATURE_FILES))
	ERRORS = $(patsubst ../testdata/%,acceptance/testdata/%.errors.ndjson,$(BAD_FEATURE_FILES))

gherkin/java/Makefile

Line 42 in 1cc2ef0

acceptance: .built $(TOKENS) $(ASTS) $(PICKLES) $(ERRORS) $(SOURCES) ## Build acceptance test dir and compare results with reference

Compare with:

gherkin/java/Makefile

Lines 13 to 17 in 1cc2ef0

	TOKENS = $(patsubst ../testdata/%,acceptance/testdata/%.tokens,$(GOOD_FEATURE_FILES))
	ASTS = $(patsubst ../testdata/%,acceptance/testdata/%.ast.ndjson,$(GOOD_FEATURE_FILES))
	PICKLES = $(patsubst ../testdata/%,acceptance/testdata/%.pickles.ndjson,$(GOOD_FEATURE_FILES))
	SOURCES = $(patsubst ../testdata/%,acceptance/testdata/%.source.ndjson,$(GOOD_FEATURE_FILES))
	ERRORS = $(patsubst ../testdata/%,acceptance/testdata/%.errors.ndjson,$(BAD_FEATURE_FILES))

gherkin/java/Makefile

Line 42 in 1cc2ef0

acceptance: .built $(TOKENS) $(ASTS) $(PICKLES) $(ERRORS) $(SOURCES) ## Build acceptance test dir and compare results with reference

This is a problem because:

As of May 2016 Gherkin is implemented in 8 languages. This number is likely to increase to a dozen within a year or two. Very few programmers are intimate with all those languages. Therefore, in order to make it as easy as possible to refactor, fix bugs, add features and release packages it is essential that all implementations have a similar structure.

CONTRIBUTING.md - consistency between implementations

✨ Do you have a proposal for making it better?

Include the tokens in the acceptance tests again.

📚 Any additional context?

The tokens were removed with cucumber/common#448 and never put back once reverted.

[mpkorstanje]

@aslakhellesoy can you pick this up?