On 2023-08-10, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> On Thursday, 10 August 2023 at 13:21:12 UTC+1, Bart wrote:
>> Searching backwards from the token's offset, determines the line-number,
>> and offset with the line.
>>
> There's a strong case for doing that. If a file is read in as a string, then any IO
> erros can be dealt with in one place.

Only if you have lexical, syntax, type, semantic and all other error
checking that needs a line number all in one place, in one pass.

On Thursday, 10 August 2023 at 16:40:40 UTC+1, Kaz Kylheku wrote:
> On 2023-08-10, Malcolm McLean <malcolm.ar...@gmail.com> wrote:
> > On Thursday, 10 August 2023 at 13:21:12 UTC+1, Bart wrote:
> >> Searching backwards from the token's offset, determines the line-number,
> >> and offset with the line.
> >>
> > There's a strong case for doing that. If a file is read in as a string, then any IO
> > erros can be dealt with in one place.
> Only if you have lexical, syntax, type, semantic and all other error
> checking that needs a line number all in one place, in one pass.
>
The "lexer" is fgetc(). You can tell because the code makes large numbers of ungetc()
calls. I'm rewriting it any way. Maybe I should do a complete rewrite to have a
proper lexer with a "gettoken()" and "match()" function. Then, as you say, all calls
to fgetc() will be in the same place, and it becomes easy to count lines and columns.

The main problem with a lexer for XML is that the grammar specifies "anything except
a special characer" for the data between nodes. So either you have to have a special
mode, so the lexer isn't really a lexer any more, or the tokens have to be single characters
anyway.

IO errors are of a different nature to parse errors. By far the most likely IO error is
failure to open the file because the wrong name was provided. But any call to
fgetc() can return EOF. If you load all the data as a string in a first pass, then
you pick up any IO errors, and further ones are not possible.

On Thu, 10 Aug 2023 15:38:47 -0000 (UTC)
Kaz Kylheku <864-117-4973@kylheku.com> wrote:
> On 2023-08-10, Malcolm McLean <malcolm.arthur.mclean@gmail.com> wrote:
> > Just adding some better error reporting to my XML parser.
> >
> > I started out by keeping track of the line number as I went. But of
> > course that's very fiddly.
>
> It's only slightly fiddly if you have to recognize tokens that can
> contain embedded newlines.

I load the whole file into memory and , as I do so , I also recognise ends of
lines and create an array where line_numbers[i] = j means that the line
number i starts at byte j .So I can get from a byte number to a line number
by a binary search. All further parsing is done after I have the whole file
in memory.

> In a language in which a newline only occurs in a comment, or
> whitespace, you can have a couple of dedicated tokenizing rules
> which increment the line number.
>
> You attach the current line number to all token objects and then
> the parser can just refer to them.
>
> If you allow tokens with newlines like
>
> "multi
> line
> literal"
>
> by a single token match that gobbles newlines, you then need
> some slight fiddling, like walking the lexeme of that object after
> it is extracted and bumping the line number for each newline
> that occurs.

[...]