Skip to content

FixedLengthTokenizer in strict mode rejects lines when the open-ended range starts at or before the first range's min #5518

Description

@harrisleesh

Bug description
The Javadoc of FixedLengthTokenizer#setColumns states: "If the last range is open, then the rest of the line is read into that column (irrespective of the strict flag setting)." However, calculateMaxRange only marks the tokenizer as open when the unbounded range's min is strictly greater than the running max seen so far:

upperBound = range.getMin();
if (upperBound > maxRange) {
    open = true;
}

Since maxRange is initialized to ranges[0].getMin(), a single open-ended column — setColumns(new Range(1)), i.e. "read the whole line into one field" — never sets open, and any range order where the unbounded range's min is not greater than the max seen so far (ranges are documented as order-independent, e.g. new Range(11), new Range(1, 10)) has the same problem. Longer lines then fail with IncorrectLineLengthException in strict mode even though the open range should absorb the rest of the line.

Environment
Spring Batch main at cb69762 (6.0.x), also reproducible on 5.x. JDK 21.

Steps to reproduce

FixedLengthTokenizer tokenizer = new FixedLengthTokenizer();
tokenizer.setColumns(new Range(1));
tokenizer.tokenize("H1        12345678       1234567890");
// IncorrectLineLengthException: Line is longer than max range 1

Expected behavior
The open-ended range absorbs the rest of the line regardless of where the unbounded range's min falls relative to the other ranges, as documented. I will submit a PR with a fix and tests.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions