This was prompted after trying to run an app on CP/M after successfully
doing so on MS-DOS. The former was much slower than I had expected.
The only bottle-neck I could see was READ-LINE. The solution was to
read the input file in one go (or at least in large chunks) and parse
out the lines, joining partial ones when they occurred. The code makes
use of library functions. I've attempted to explain these for anyone
thinking of porting it.

https://pastebin.com/XpBmTFXW

In article <ua35nq$2kjcn$1@dont-email.me>, dxforth <dxforth@gmail.com> wrote:
>This was prompted after trying to run an app on CP/M after successfully
>doing so on MS-DOS. The former was much slower than I had expected.
>The only bottle-neck I could see was READ-LINE. The solution was to
>read the input file in one go (or at least in large chunks) and parse
>out the lines, joining partial ones when they occurred. The code makes
>use of library functions. I've attempted to explain these for anyone
>thinking of porting it.
>
>https://pastebin.com/XpBmTFXW

As long as you are buffering you can deal out the lines without
copying as a string constant (addr n).
I called this (ACCEPT). This does away with the need to provide
a buffer. You must not of course change the string constant
returned.

(ACCEPT)
STACKEFFECT: -- sc
Accept characters from the terminal, until a RET is received and
return the result as a constant string sc. It doesn't contain any line
ending, but the buffer still does and after 1+ the string ends in a
LF. The editing functions are the same as with ACCEPT . This is
lighter on the system and sometimes easier to use than ACCEPT This
input remains valid until the next time that the console buffer is
refilled.
--
Don't praise the day before the evening. One swallow doesn't make spring.
You must not say "hey" before you have crossed the bridge. Don't sell the
hide of the bear until you shot it. Better one bird in the hand than ten in
the air. First gain is a cat spinning. - the Wise from Antrim -

On Saturday, July 29, 2023 at 7:11:27 PM UTC-4, none albert wrote:
> In article <ua35nq$2kjcn$1...@dont-email.me>, dxforth <dxf...@gmail.com> wrote:
> >This was prompted after trying to run an app on CP/M after successfully
> >doing so on MS-DOS. The former was much slower than I had expected.
> >The only bottle-neck I could see was READ-LINE. The solution was to
> >read the input file in one go (or at least in large chunks) and parse
> >out the lines, joining partial ones when they occurred. The code makes
> >use of library functions. I've attempted to explain these for anyone
> >thinking of porting it.
> >
> >https://pastebin.com/XpBmTFXW
> As long as you are buffering you can deal out the lines without
> copying as a string constant (addr n).
> I called this (ACCEPT). This does away with the need to provide
> a buffer. You must not of course change the string constant
> returned.
>
> (ACCEPT)
> STACKEFFECT: -- sc
> Accept characters from the terminal, until a RET is received and
> return the result as a constant string sc. It doesn't contain any line
> ending, but the buffer still does and after 1+ the string ends in a
> LF. The editing functions are the same as with ACCEPT . This is
> lighter on the system and sometimes easier to use than ACCEPT This
> input remains valid until the next time that the console buffer is
> refilled.

How is "RET" defined? "Terminal" is not by necessity the keyboard of a computer, right? So what ASCII sequence is equated to "RET"?

--

Rick C.

- Get 1,000 miles of free Supercharging
- Tesla referral code - https://ts.la/richard11209

On 30/07/2023 9:11 am, albert wrote:
> ...
> As long as you are buffering you can deal out the lines without
> copying as a string constant (addr n).

AFAICS that's only feasable if the whole file fits in memory. I don't
have that option.

The main issue in my case seems to be the file-repositioning that is
the basis of most READ-LINE implementations. Under CP/M it hits hard
for some reason. I should first verify the algorithm I used is as
efficient as can be. (It came from a C compiler, so how could it not? :)

dxforth <dxforth@gmail.com> writes:
>The main issue in my case seems to be the file-repositioning that is
>the basis of most READ-LINE implementations. Under CP/M it hits hard
>for some reason.

Not just on CP/M. E.g., on Linux I reported
<2021Oct19.095538@mips.complang.tuwien.ac.at>:

|perf stat -e cycles:u -e instructions:u ~/gforth/gforth -e '"count-unique.in" r/o open-file throw constant f create buf 256 allot : foo 0 begin buf 256 f read-line throw nip while 1+ repeat ; foo . cr bye'
| |count-unique.in is the input for Ben Hoyt's count-unique example and
|consists of 10 bibles, 998170 lines, or 43MB. On a 4GHz Skylake
|gforth-fast takes 802M cycles or 0.201s user time for this (and 31M
|cycles or 0.011s without the call to FOO).
| |[...]
|I have also measured SwiftForth 3.11.0 with the
|same benchmark:
| |[...]
| 5,945,307,655 cycles
| 768,834,475 cycles:u
| 5,131,957,800 cycles:k
| 733,948,005 instructions:u # 0.95 insn per cycle
| | 1.488727585 seconds time elapsed
| | 0.916473000 seconds user
| 0.572295000 seconds sys

I.e., this just measures reading a 43MB file with 998170 lines with
repeated READ-LINEs. Gforth uses buffered I/O (with the buffering
implemented by glibc), while SwiftForth uses unbuffered I/O and
REPOSITION-FILE. As a result, SwiftForth spends more than 7 times
more cycles on this benchmark than Gforth. I would be surprised if
the strategy used by SwiftForth fared better than buffering on other
OSs.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: https://forth-standard.org/
EuroForth 2023: https://euro.theforth.net/2023

On 30/07/2023 3:32 pm, Anton Ertl wrote:
> ...
> I.e., this just measures reading a 43MB file with 998170 lines with
> repeated READ-LINEs. Gforth uses buffered I/O (with the buffering
> implemented by glibc), while SwiftForth uses unbuffered I/O and
> REPOSITION-FILE. As a result, SwiftForth spends more than 7 times
> more cycles on this benchmark than Gforth. I would be surprised if
> the strategy used by SwiftForth fared better than buffering on other
> OSs.

For INCLUDED et al, SwiftForth reads the entire file. For some reason
they use a different EOL scanner than is used for READ-LINE.

Further experiments with CP/M showed I didn't need to read all or most
of the file to gain a substantial speed increase. A 512 byte buffer
was sufficient to gain a 10x improvement over READ-LINE. Raising it
50KB produced no noticeable benefit. /EOL was written in assembler.

In article <ua4id9$2s1s9$1@dont-email.me>, dxforth <dxforth@gmail.com> wrote:
>On 30/07/2023 9:11 am, albert wrote:
>> ...
>> As long as you are buffering you can deal out the lines without
>> copying as a string constant (addr n).
>
>AFAICS that's only feasable if the whole file fits in memory. I don't
>have that option.

I invented (ACCEPT) that exactly for the situation reading from a stream.
(I never ever use READ-LINE for a file that fits in memory.)
It goes as follows:
1.
If there is a ret after the pointer in the input buffer,
return the part between the pointer and the ret
advance pointer. you're done
2.
copy the remainder to the start of the buffer
read from the sream to fill the buffer
continue at step 1

And for Rick : RET is trimmed off, so why don you care?

>The main issue in my case seems to be the file-repositioning that is
>the basis of most READ-LINE implementations. Under CP/M it hits hard
>for some reason. I should first verify the algorithm I used is as
>efficient as can be. (It came from a C compiler, so how could it not? :)

Under CP/M file segments are visible. I would read only whole segments
which made the above algorithm even simpler. The copying step
is superfluous.

Groetjes Albert
--
Don't praise the day before the evening. One swallow doesn't make spring.
You must not say "hey" before you have crossed the bridge. Don't sell the
hide of the bear until you shot it. Better one bird in the hand than ten in
the air. First gain is a cat spinning. - the Wise from Antrim -

In article <2023Jul30.073239@mips.complang.tuwien.ac.at>,
Anton Ertl <anton@mips.complang.tuwien.ac.at> wrote:
>dxforth <dxforth@gmail.com> writes:
>>The main issue in my case seems to be the file-repositioning that is
>>the basis of most READ-LINE implementations. Under CP/M it hits hard
>>for some reason.
>
>Not just on CP/M. E.g., on Linux I reported
><2021Oct19.095538@mips.complang.tuwien.ac.at>:
>
>|perf stat -e cycles:u -e instructions:u ~/gforth/gforth -e
>'"count-unique.in" r/o open-file throw constant f create buf 256 allot :
>foo 0 begin buf 256 f read-line throw nip while 1+ repeat ; foo . cr
>bye'
>|
>|count-unique.in is the input for Ben Hoyt's count-unique example and
>|consists of 10 bibles, 998170 lines, or 43MB. On a 4GHz Skylake
>|gforth-fast takes 802M cycles or 0.201s user time for this (and 31M
>|cycles or 0.011s without the call to FOO).
>|
>|[...]
>|I have also measured SwiftForth 3.11.0 with the
>|same benchmark:
>|
>|[...]
>| 5,945,307,655 cycles
>| 768,834,475 cycles:u
>| 5,131,957,800 cycles:k
>| 733,948,005 instructions:u # 0.95 insn per cycle
>|
>| 1.488727585 seconds time elapsed
>|
>| 0.916473000 seconds user
>| 0.572295000 seconds sys
>
>I.e., this just measures reading a 43MB file with 998170 lines with
>repeated READ-LINEs. Gforth uses buffered I/O (with the buffering
>implemented by glibc), while SwiftForth uses unbuffered I/O and
>REPOSITION-FILE. As a result, SwiftForth spends more than 7 times
>more cycles on this benchmark than Gforth. I would be surprised if
>the strategy used by SwiftForth fared better than buffering on other
>OSs.

I now possess a workstation with 256 Gbyte. Not slurping a mere
43 Mbyte file seems ludicrous.
Was there ever a CP/M system with 43 Mbyte disk storage?

(The results are interesting, nevertheless, but hardly surprising.)

>- anton
--
Don't praise the day before the evening. One swallow doesn't make spring.
You must not say "hey" before you have crossed the bridge. Don't sell the
hide of the bear until you shot it. Better one bird in the hand than ten in
the air. First gain is a cat spinning. - the Wise from Antrim -

On 30/07/2023 7:29 pm, albert wrote:
> In article <ua4id9$2s1s9$1@dont-email.me>, dxforth <dxforth@gmail.com> wrote:
>> On 30/07/2023 9:11 am, albert wrote:
>>> ...
>>> As long as you are buffering you can deal out the lines without
>>> copying as a string constant (addr n).
>>
>> AFAICS that's only feasable if the whole file fits in memory. I don't
>> have that option.
>
> I invented (ACCEPT) that exactly for the situation reading from a stream.
> (I never ever use READ-LINE for a file that fits in memory.)
> It goes as follows:
> 1.
> If there is a ret after the pointer in the input buffer,
> return the part between the pointer and the ret
> advance pointer. you're done
> 2.
> copy the remainder to the start of the buffer
> read from the sream to fill the buffer
> continue at step 1

That crossed my mind.

>> The main issue in my case seems to be the file-repositioning that is
>> the basis of most READ-LINE implementations. Under CP/M it hits hard
>> for some reason. I should first verify the algorithm I used is as
>> efficient as can be. (It came from a C compiler, so how could it not? :)
>
> Under CP/M file segments are visible. I would read only whole segments
> which made the above algorithm even simpler. The copying step
> is superfluous.

That I didn't consider ... just because a buffer is a given size, doesn't
mean one has to fill it. In any case, I'm back to using a small buffer
and the copying appears to have negligible impact.

On Sunday, July 30, 2023 at 6:26:56 AM UTC-5, dxforth wrote:

I do the same as Albert, read in a buffer then point to a
line and work down the buffer till no more line feed then
move the residue up and do again.
Back in DOS days had heard that 4096 byte buffer was
optimum for the read so that was what I used but I also
used much smaller buffer without problem.
This is still on a line basis. So I also had a stream
input for things like production HTML which has no line
ending.
Note also that Jones Forth inputs on a word basis not line,
why no OK prompt, and should have no problem with steams.
--
me

On Sunday, July 30, 2023 at 5:29:24 AM UTC-4, none albert wrote:
> In article <ua4id9$2s1s9$1...@dont-email.me>, dxforth <dxf...@gmail.com> wrote:
> >On 30/07/2023 9:11 am, albert wrote:
> >> ...
> >> As long as you are buffering you can deal out the lines without
> >> copying as a string constant (addr n).
> >
> >AFAICS that's only feasable if the whole file fits in memory. I don't
> >have that option.
> I invented (ACCEPT) that exactly for the situation reading from a stream.
> (I never ever use READ-LINE for a file that fits in memory.)
> It goes as follows:
> 1.
> If there is a ret after the pointer in the input buffer,
> return the part between the pointer and the ret
> advance pointer. you're done
> 2.
> copy the remainder to the start of the buffer
> read from the sream to fill the buffer
> continue at step 1
>
> And for Rick : RET is trimmed off, so why don you care?

Because it has to be sent to be recognized. The specification for RET determines what is sent from the terminal.

This is not purely about the program. Programs have to live in a world of things. There are a lot more devices using <$10 MCUs than there are >$100 CPUs.

--

Rick C.

+ Get 1,000 miles of free Supercharging
+ Tesla referral code - https://ts.la/richard11209

On 31/07/2023 12:41 am, S Jack wrote:
> On Sunday, July 30, 2023 at 6:26:56 AM UTC-5, dxforth wrote:
>
> I do the same as Albert, read in a buffer then point to a
> line and work down the buffer till no more line feed then
> move the residue up and do again.
> Back in DOS days had heard that 4096 byte buffer was
> optimum for the read so that was what I used but I also
> used much smaller buffer without problem.

What happens should a string exceed the buffer size? Mine keeps
working but has the opposite problem of potentially overwriting
memory if there's no length check.

On 30/07/2023 5:07 pm, dxforth wrote:
> ...
> Further experiments with CP/M showed I didn't need to read all or most
> of the file to gain a substantial speed increase. A 512 byte buffer
> was sufficient to gain a 10x improvement over READ-LINE. Raising it
> 50KB produced no noticeable benefit. /EOL was written in assembler.

Unfortunately it has a bug. If the last byte in the buffer is the
first char of a CRLF it will result in an additional empty line.
Initial tests with a large buffer didn't reveal the problem. The EOL
scanner itself is fine. It simply wasn't intended for use in a stream
which doesn't backtrack.

In article <uaa03r$3ijne$1@dont-email.me>, dxforth <dxforth@gmail.com> wrote:
>On 30/07/2023 5:07 pm, dxforth wrote:
>> ...
>> Further experiments with CP/M showed I didn't need to read all or most
>> of the file to gain a substantial speed increase. A 512 byte buffer
>> was sufficient to gain a 10x improvement over READ-LINE. Raising it
>> 50KB produced no noticeable benefit. /EOL was written in assembler.
>
>Unfortunately it has a bug. If the last byte in the buffer is the
>first char of a CRLF it will result in an additional empty line.
>Initial tests with a large buffer didn't reveal the problem. The EOL
>scanner itself is fine. It simply wasn't intended for use in a stream
>which doesn't backtrack.
>
Use a circular buffer array that corresponds to disk sectors.
If you discover that you can't do READ-LINE because the line endings
are not present in the buffers, read the buffers that have become
empty. Then try again.
How long the line endings are doesn't matter.

Groetjes Albert
--
Don't praise the day before the evening. One swallow doesn't make spring.
You must not say "hey" before you have crossed the bridge. Don't sell the
hide of the bear until you shot it. Better one bird in the hand than ten in
the air. First gain is a cat spinning. - the Wise from Antrim -

On 1/08/2023 6:43 pm, albert wrote:
> In article <uaa03r$3ijne$1@dont-email.me>, dxforth <dxforth@gmail.com> wrote:
>> On 30/07/2023 5:07 pm, dxforth wrote:
>>> ...
>>> Further experiments with CP/M showed I didn't need to read all or most
>>> of the file to gain a substantial speed increase. A 512 byte buffer
>>> was sufficient to gain a 10x improvement over READ-LINE. Raising it
>>> 50KB produced no noticeable benefit. /EOL was written in assembler.
>>
>> Unfortunately it has a bug. If the last byte in the buffer is the
>> first char of a CRLF it will result in an additional empty line.
>> Initial tests with a large buffer didn't reveal the problem. The EOL
>> scanner itself is fine. It simply wasn't intended for use in a stream
>> which doesn't backtrack.
>>
> Use a circular buffer array that corresponds to disk sectors.
> If you discover that you can't do READ-LINE because the line endings
> are not present in the buffers, read the buffers that have become
> empty. Then try again.
> How long the line endings are doesn't matter.

In the end I used a buffered structure I previously implemented for
reading files a byte at a time. All that was needed was turning the
bytes into lines. Usage is similar to READ-LINE.