Add non-validating edi segment reader, sample and doc (#133)

Author: jf-tech

README.md

@@ -64,6 +64,7 @@ situations.
 - Golang 1.14
 
 ## Recent Major Feature Additions/Changes
+- Added `NonValidatingReader` EDI segment reader.
 - Added fixed-length file format support in omniv21 handler.
 - Added EDI file format support in omniv21 handler.
 - Major restructure/refactoring

doc/programmability.md

@@ -1,19 +1,20 @@
 * [Programmability of Omniparser](#programmability-of-omniparser)
-    * [Out\-of\-Box Basic Use Case](#out-of-box-basic-use-case)
-    * [Add A New custom\_func](#add-a-new-custom_func)
-    * [Add A New custom\_parse](#add-a-new-custom_parse)
-    * [Add A New File Format](#add-a-new-file-format)
-    * [Add A New Schema Handler](#add-a-new-schema-handler)
-    * [Put All Together](#put-all-together)
-    * [In Non\-Golang Environment](#in-non-golang-environment)
+  * [Out\-of\-Box Basic Use Case](#out-of-box-basic-use-case)
+  * [Add A New custom\_func](#add-a-new-custom_func)
+  * [Add A New custom\_parse](#add-a-new-custom_parse)
+  * [Add A New File Format](#add-a-new-file-format)
+  * [Add A New Schema Handler](#add-a-new-schema-handler)
+  * [Put All Together](#put-all-together)
+  * [In Non\-Golang Environment](#in-non-golang-environment)
 * [Programmability of Some Components without Omniparser](#programmability-of-some-components-without-omniparser)
-    * [Functions](#functions)
-    * [IDR](#idr)
-    * [CSV Reader](#csv-reader)
-    * [Fixed\-Length Reader](#fixed-length-reader)
-    * [EDI Reader](#edi-reader)
-    * [JSON Reader](#json-reader)
-    * [XML Reader](#xml-reader)
+  * [Functions](#functions)
+  * [IDR](#idr)
+  * [CSV Reader](#csv-reader)
+  * [Fixed\-Length Reader](#fixed-length-reader)
+  * [Full EDI Reader](#full-edi-reader)
+  * [Non\-Validating EDI Segment Reader](#non-validating-edi-segment-reader)
+  * [JSON Reader](#json-reader)
+  * [XML Reader](#xml-reader)
 
 # Programmability of Omniparser
 
@@ -237,15 +238,42 @@ reader that does
 For more reader specific settings/configurations, check
 [Fixed-Length in Depth](./fixedlength_in_depth.md) page.
 
-## EDI Reader
+## Full EDI Reader
 
 Use [`NewReader()`](../extensions/omniv21/fileformat/edi/reader.go) to create an EDI reader that does
 - segment min/max validation
 - XPath based data row filtering
 - Context-aware error message
 
-Future TO-DO: create a version of non-validating EDI reader for users who are only interested in
-getting the raw segment data, without any validation.
+For more reader specific settings/configurations, check
+[EDI in Depth](./edi_in_depth.md) page.
+
+## Non-Validating EDI Segment Reader
+
+Use [`NewNonValidatingReader()`](../extensions/omniv21/fileformat/edi/reader2.go) to create a
+non-validating EDI segment reader. Sometimes user might not want the full EDI reader that does
+many packing/unpacking and structural/hierarchical validations, they simply need an EDI segment
+reader that reads out all the raw segments and their elements/components.
+
+Usage example:
+```
+r := edi.NewNonValidatingReader(
+    strings.NewReader("....."),
+    &edi.FileDecl{
+        SegDelim: ...,
+        ElemDelim: ...,
+        ...,
+        // No need to set SegDecls. Just all the needed top level edi.FileDecl settings.
+    })
+for {
+    seg, err := r.Read()
+    if err == io.EOF {
+        break
+    }
+    if err != nil { ... }
+    // seg contains the raw segment data, and is of edi.RawSeg type.
+}
+```
 
 ## JSON Reader
 See [IDR](#idr) notes about the JSON/XML readers above.

doc/use_of_custom_funcs.md

@@ -1,8 +1,8 @@
 # Use of `custom_func`, Specially `javascript`
 
 `custom_func` is a transform that allows schema writer to alter, compose, transform and aggregate existing
-data from the input. Among [all `custom_func`](./customfuncs.md), [`javascript`](TODO) is the most important
-one to understand and master.
+data from the input. Among [all `custom_func`](./customfuncs.md),
+[`javascript`](./customfuncs.md#javascript) is the most important one to understand and master.
 
 A `custom_func` has 4 basic parts: `xpath`/`xpath_dynamic`, `name`, `args`, and `type`.
 

extensions/omniv21/fileformat/edi/bench_test.go

@@ -9,7 +9,7 @@ import (
 	"github.com/jf-tech/go-corelib/testlib"
 )
 
-// Adding a benchmark for rawSeg operation to ensure there is no alloc:
+// Adding a benchmark for RawSeg operation to ensure there is no alloc:
 // BenchmarkRawSeg-8   	81410766	        13.9 ns/op	       0 B/op	       0 allocs/op
 func BenchmarkRawSeg(b *testing.B) {
 	rawSegName := "test"
@@ -20,11 +20,11 @@ func BenchmarkRawSeg(b *testing.B) {
 	for i := 0; i < b.N; i++ {
 		r.resetRawSeg()
 		r.unprocessedRawSeg.valid = true
-		r.unprocessedRawSeg.name = rawSegName
-		r.unprocessedRawSeg.raw = rawSegData
-		r.unprocessedRawSeg.elems = append(
-			r.unprocessedRawSeg.elems,
-			rawSegElem{1, 1, rawSegData[0:4], false}, rawSegElem{2, 1, rawSegData[5:], false})
+		r.unprocessedRawSeg.Name = rawSegName
+		r.unprocessedRawSeg.Raw = rawSegData
+		r.unprocessedRawSeg.Elems = append(
+			r.unprocessedRawSeg.Elems,
+			RawSegElem{1, 1, rawSegData[0:4]}, RawSegElem{2, 1, rawSegData[5:]})
 	}
 }
 
@@ -534,16 +534,16 @@ func BenchmarkGetUnprocessedRawSeg_WithCompAndRelease(b *testing.B) {
 }
 
 var (
-	benchRawSegToNodeRawSeg = rawSeg{
+	benchRawSegToNodeRawSeg = RawSeg{
 		valid: true,
-		name:  "ISA",
-		raw:   []byte("ISA*0*1:2*3*"),
-		elems: []rawSegElem{
-			{0, 1, []byte("ISA"), false},
-			{1, 1, []byte("0"), false},
-			{2, 1, []byte("1"), false},
-			{2, 2, []byte("2"), false},
-			{3, 1, []byte("3"), false},
+		Name:  "ISA",
+		Raw:   []byte("ISA*0*1:2*3*"),
+		Elems: []RawSegElem{
+			{0, 1, []byte("ISA")},
+			{1, 1, []byte("0")},
+			{2, 1, []byte("1")},
+			{2, 2, []byte("2")},
+			{3, 1, []byte("3")},
 		},
 	}
 	benchRawSegToNodeDecl = &SegDecl{
@@ -555,9 +555,9 @@ var (
 		},
 		fqdn: "ISA",
 	}
-	// we can do this (reusing reader and its rawSeg again & again in benchmark
+	// we can do this (reusing reader and its RawSeg again & again in benchmark
 	// because there is no release-char thus there is no data modification in
-	// rawSeg.elems
+	// RawSeg.Elems
 	benchRawSegToNodeReader = &ediReader{unprocessedRawSeg: benchRawSegToNodeRawSeg}
 )