Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 4c56223

Browse filesBrowse files
jf-techjf-tech
authored
Add non-validating edi segment reader, sample and doc (#133)
1 parent df96f55 commit 4c56223
Copy full SHA for 4c56223

File tree

Expand file treeCollapse file tree

12 files changed

+46315
-313
lines changed
Filter options
Expand file treeCollapse file tree

12 files changed

+46315
-313
lines changed

‎README.md

Copy file name to clipboardExpand all lines: README.md
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ situations.
6464
- Golang 1.14
6565

6666
## Recent Major Feature Additions/Changes
67+
- Added `NonValidatingReader` EDI segment reader.
6768
- Added fixed-length file format support in omniv21 handler.
6869
- Added EDI file format support in omniv21 handler.
6970
- Major restructure/refactoring

‎doc/programmability.md

Copy file name to clipboardExpand all lines: doc/programmability.md
+45-17Lines changed: 45 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,20 @@
11
* [Programmability of Omniparser](#programmability-of-omniparser)
2-
* [Out\-of\-Box Basic Use Case](#out-of-box-basic-use-case)
3-
* [Add A New custom\_func](#add-a-new-custom_func)
4-
* [Add A New custom\_parse](#add-a-new-custom_parse)
5-
* [Add A New File Format](#add-a-new-file-format)
6-
* [Add A New Schema Handler](#add-a-new-schema-handler)
7-
* [Put All Together](#put-all-together)
8-
* [In Non\-Golang Environment](#in-non-golang-environment)
2+
* [Out\-of\-Box Basic Use Case](#out-of-box-basic-use-case)
3+
* [Add A New custom\_func](#add-a-new-custom_func)
4+
* [Add A New custom\_parse](#add-a-new-custom_parse)
5+
* [Add A New File Format](#add-a-new-file-format)
6+
* [Add A New Schema Handler](#add-a-new-schema-handler)
7+
* [Put All Together](#put-all-together)
8+
* [In Non\-Golang Environment](#in-non-golang-environment)
99
* [Programmability of Some Components without Omniparser](#programmability-of-some-components-without-omniparser)
10-
* [Functions](#functions)
11-
* [IDR](#idr)
12-
* [CSV Reader](#csv-reader)
13-
* [Fixed\-Length Reader](#fixed-length-reader)
14-
* [EDI Reader](#edi-reader)
15-
* [JSON Reader](#json-reader)
16-
* [XML Reader](#xml-reader)
10+
* [Functions](#functions)
11+
* [IDR](#idr)
12+
* [CSV Reader](#csv-reader)
13+
* [Fixed\-Length Reader](#fixed-length-reader)
14+
* [Full EDI Reader](#full-edi-reader)
15+
* [Non\-Validating EDI Segment Reader](#non-validating-edi-segment-reader)
16+
* [JSON Reader](#json-reader)
17+
* [XML Reader](#xml-reader)
1718

1819
# Programmability of Omniparser
1920

@@ -237,15 +238,42 @@ reader that does
237238
For more reader specific settings/configurations, check
238239
[Fixed-Length in Depth](./fixedlength_in_depth.md) page.
239240
240-
## EDI Reader
241+
## Full EDI Reader
241242
242243
Use [`NewReader()`](../extensions/omniv21/fileformat/edi/reader.go) to create an EDI reader that does
243244
- segment min/max validation
244245
- XPath based data row filtering
245246
- Context-aware error message
246247
247-
Future TO-DO: create a version of non-validating EDI reader for users who are only interested in
248-
getting the raw segment data, without any validation.
248+
For more reader specific settings/configurations, check
249+
[EDI in Depth](./edi_in_depth.md) page.
250+
251+
## Non-Validating EDI Segment Reader
252+
253+
Use [`NewNonValidatingReader()`](../extensions/omniv21/fileformat/edi/reader2.go) to create a
254+
non-validating EDI segment reader. Sometimes user might not want the full EDI reader that does
255+
many packing/unpacking and structural/hierarchical validations, they simply need an EDI segment
256+
reader that reads out all the raw segments and their elements/components.
257+
258+
Usage example:
259+
```
260+
r := edi.NewNonValidatingReader(
261+
strings.NewReader("....."),
262+
&edi.FileDecl{
263+
SegDelim: ...,
264+
ElemDelim: ...,
265+
...,
266+
// No need to set SegDecls. Just all the needed top level edi.FileDecl settings.
267+
})
268+
for {
269+
seg, err := r.Read()
270+
if err == io.EOF {
271+
break
272+
}
273+
if err != nil { ... }
274+
// seg contains the raw segment data, and is of edi.RawSeg type.
275+
}
276+
```
249277
250278
## JSON Reader
251279
See [IDR](#idr) notes about the JSON/XML readers above.

‎doc/use_of_custom_funcs.md

Copy file name to clipboardExpand all lines: doc/use_of_custom_funcs.md
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Use of `custom_func`, Specially `javascript`
22

33
`custom_func` is a transform that allows schema writer to alter, compose, transform and aggregate existing
4-
data from the input. Among [all `custom_func`](./customfuncs.md), [`javascript`](TODO) is the most important
5-
one to understand and master.
4+
data from the input. Among [all `custom_func`](./customfuncs.md),
5+
[`javascript`](./customfuncs.md#javascript) is the most important one to understand and master.
66

77
A `custom_func` has 4 basic parts: `xpath`/`xpath_dynamic`, `name`, `args`, and `type`.
88

‎extensions/omniv21/fileformat/edi/bench_test.go

Copy file name to clipboardExpand all lines: extensions/omniv21/fileformat/edi/bench_test.go
+17-17Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import (
99
"github.com/jf-tech/go-corelib/testlib"
1010
)
1111

12-
// Adding a benchmark for rawSeg operation to ensure there is no alloc:
12+
// Adding a benchmark for RawSeg operation to ensure there is no alloc:
1313
// BenchmarkRawSeg-8 81410766 13.9 ns/op 0 B/op 0 allocs/op
1414
func BenchmarkRawSeg(b *testing.B) {
1515
rawSegName := "test"
@@ -20,11 +20,11 @@ func BenchmarkRawSeg(b *testing.B) {
2020
for i := 0; i < b.N; i++ {
2121
r.resetRawSeg()
2222
r.unprocessedRawSeg.valid = true
23-
r.unprocessedRawSeg.name = rawSegName
24-
r.unprocessedRawSeg.raw = rawSegData
25-
r.unprocessedRawSeg.elems = append(
26-
r.unprocessedRawSeg.elems,
27-
rawSegElem{1, 1, rawSegData[0:4], false}, rawSegElem{2, 1, rawSegData[5:], false})
23+
r.unprocessedRawSeg.Name = rawSegName
24+
r.unprocessedRawSeg.Raw = rawSegData
25+
r.unprocessedRawSeg.Elems = append(
26+
r.unprocessedRawSeg.Elems,
27+
RawSegElem{1, 1, rawSegData[0:4]}, RawSegElem{2, 1, rawSegData[5:]})
2828
}
2929
}
3030

@@ -534,16 +534,16 @@ func BenchmarkGetUnprocessedRawSeg_WithCompAndRelease(b *testing.B) {
534534
}
535535

536536
var (
537-
benchRawSegToNodeRawSeg = rawSeg{
537+
benchRawSegToNodeRawSeg = RawSeg{
538538
valid: true,
539-
name: "ISA",
540-
raw: []byte("ISA*0*1:2*3*"),
541-
elems: []rawSegElem{
542-
{0, 1, []byte("ISA"), false},
543-
{1, 1, []byte("0"), false},
544-
{2, 1, []byte("1"), false},
545-
{2, 2, []byte("2"), false},
546-
{3, 1, []byte("3"), false},
539+
Name: "ISA",
540+
Raw: []byte("ISA*0*1:2*3*"),
541+
Elems: []RawSegElem{
542+
{0, 1, []byte("ISA")},
543+
{1, 1, []byte("0")},
544+
{2, 1, []byte("1")},
545+
{2, 2, []byte("2")},
546+
{3, 1, []byte("3")},
547547
},
548548
}
549549
benchRawSegToNodeDecl = &SegDecl{
@@ -555,9 +555,9 @@ var (
555555
},
556556
fqdn: "ISA",
557557
}
558-
// we can do this (reusing reader and its rawSeg again & again in benchmark
558+
// we can do this (reusing reader and its RawSeg again & again in benchmark
559559
// because there is no release-char thus there is no data modification in
560-
// rawSeg.elems
560+
// RawSeg.Elems
561561
benchRawSegToNodeReader = &ediReader{unprocessedRawSeg: benchRawSegToNodeRawSeg}
562562
)
563563

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.