Hi.

Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.

PDOS/386 (available at http://pdos.org) is able
to run a hello world Linux ELF binary, and recent
developments mean that I should be able to run
more than just a hello world soon.

But even with those recent developments, I am
still missing the same thing that Cygwin was
missing, and added O_TEXT to solve.

I'd like to add an O_TEXT flag to Linux. It would
be good if Linux officially reserved a flag for
this use, but in the absence of that, I'm happy
to create my own.

I think Linux adds flags going up, so I would add
any flags I want going down.

So I'm thinking of making O_TEXT 0x80000000

Another possibility would be 0x40000000 to avoid
being seen as a negative number.

This is different from Cygwin, which uses 0x20000.

I would be happy to use the Cygwin value, but I
think Linux may have already encroached that.

And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.

Any suggestions?

Thanks. Paul.

Paul Edwards <mutazilah@gmail.com> writes:
> Cygwin has an O_TEXT on the open() call to let "the OS layer" (can be
> quibbled) that the file is being opened in text mode. And as such, it
> gives that layer an opportunity to insert CRs.
>
> PDOS/386 (available at http://pdos.org) is able to run a hello world
> Linux ELF binary, and recent developments mean that I should be able
> to run more than just a hello world soon.
>
> But even with those recent developments, I am still missing the same
> thing that Cygwin was missing, and added O_TEXT to solve.

IMO Cygwin got this one wrong. The text/binary file distinction should
be managed in the C library or higher. Applications should either use
the appropriate arguments to fopen(), or manage any translation required
themselves.

> I'd like to add an O_TEXT flag to Linux. It would be good if Linux
> officially reserved a flag for this use, but in the absence of that,
> I'm happy to create my own.

You’d need to ask on the linux-kernel mailing list, I think. But I
wouldn’t predict a very positive response.

--
https://www.greenend.org.uk/rjk/

On 20/02/24 17:53, Richard Kettlewell wrote:

> Paul Edwards <mutazilah@gmail.com> writes:
>> Cygwin has an O_TEXT on the open() call to let "the OS layer" (can be
>> quibbled) that the file is being opened in text mode. And as such, it
>> gives that layer an opportunity to insert CRs.
>>
>> PDOS/386 (available at http://pdos.org) is able to run a hello world
>> Linux ELF binary, and recent developments mean that I should be able
>> to run more than just a hello world soon.
>>
>> But even with those recent developments, I am still missing the same
>> thing that Cygwin was missing, and added O_TEXT to solve.
>
> IMO Cygwin got this one wrong. The text/binary file distinction should
> be managed in the C library or higher. Applications should either use
> the appropriate arguments to fopen(), or manage any translation required
> themselves.

Hi Richard. Thanks for your reply.

The C library that I use for Linux is a Linux port
of PDPCLIB, and is designed for Linux, and thus, no
translation will be done.

It is only when I run that Linux ELF binary on
PDOS/386 (mini Win32 clone) that I want translation
done. And even that would be a PDOS/386 configuration
thing. Maybe I would like to run a NL-only system.

So it can't easily (the non-Linux environment would
need to be detected, and assumptions made) be done in
the (Linux) C library, and it wouldn't be appropriate
either.

>> I'd like to add an O_TEXT flag to Linux. It would be good if Linux
>> officially reserved a flag for this use, but in the absence of that,
>> I'm happy to create my own.
>
> You’d need to ask on the linux-kernel mailing list, I think. But I
> wouldn’t predict a very positive response.

That's what I'm predicting too. So can you suggest
a flag? I think Linux is consuming flags going up,
so I could go down for anything I need, and have
0x80000000 mean "O_TEXT".

But that makes it a negative number, which may not
be nice, so I could make it 0x40000000.

Another possibility would be to commandeer some
existing unimportant flag. I think there might be
one for "access time"? Since I don't care whether
or not access time is updated, if access time is
requested, I could make it mean "text".

Or some combination of flags.

Any suggestions within the constraints of the Linux
people having different goals to me?

Thanks. Paul.

On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:

> Hi.
>
> Cygwin has an O_TEXT on the open() call to let
> "the OS layer" (can be quibbled) that the file
> is being opened in text mode. And as such, it
> gives that layer an opportunity to insert CRs.
[snip]
> I'd like to add an O_TEXT flag to Linux.[snip]
> Any suggestions?

Yes, one. Dont.

Microsoft Windows (on top of which Cygwin must run) makes a low-level
distinction between "binary" files and "text" files. I believe it is
because Windows guarantees (or must guarantee) certain data
transformations on text that do not apply to other forms of data.
(I.e. translating <cr><lf> sequences into C newline characters, etc)

HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
and binary data, neither in the standard I/O library, nor in the OS
itself.

You are effectively asking that Linux /NOW/ make such a distinction,
even if only to ignore the O_TEXT flag that you ask for.

It ain't gonna fly. Such a flag would invalidate /decades/ worth of
Unix/Linux code, and /not/ provide any functionality either to
the OS layer, the stdio library layer, or the application code layer.

So, dont even try to "add an O_TEXT flag" to Linux. Just dont.
--
Lew Pitcher
"In Skills We Trust"

On 20/02/24 22:14, Lew Pitcher wrote:
> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>
>> Hi.
>>
>> Cygwin has an O_TEXT on the open() call to let
>> "the OS layer" (can be quibbled) that the file
>> is being opened in text mode. And as such, it
>> gives that layer an opportunity to insert CRs.
> [snip]
>> I'd like to add an O_TEXT flag to Linux.[snip]
>> Any suggestions?
>
> Yes, one. Dont.
>
> Microsoft Windows (on top of which Cygwin must run) makes a low-level
> distinction between "binary" files and "text" files. I believe it is
> because Windows guarantees (or must guarantee) certain data
> transformations on text that do not apply to other forms of data.
> (I.e. translating <cr><lf> sequences into C newline characters, etc)
>
> HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
> and binary data, neither in the standard I/O library, nor in the OS
> itself.
>
> You are effectively asking that Linux /NOW/ make such a distinction,
> even if only to ignore the O_TEXT flag that you ask for.

The C library that most people use already has
that distinction - and it is indeed ignored.
Works perfectly fine.

> It ain't gonna fly. Such a flag would invalidate /decades/ worth of
> Unix/Linux code, and /not/ provide any functionality either to
> the OS layer, the stdio library layer, or the application code layer.

Sorry - what is being invalidated? Can you give an
example?

BFN. Paul.

Lew Pitcher <lew.pitcher@digitalfreehold.ca> writes:
> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>> Cygwin has an O_TEXT on the open() call to let
>> "the OS layer" (can be quibbled) that the file
>> is being opened in text mode. And as such, it
>> gives that layer an opportunity to insert CRs.
> [snip]
>> I'd like to add an O_TEXT flag to Linux.[snip]
>> Any suggestions?
>
> Yes, one. Dont.
>
> Microsoft Windows (on top of which Cygwin must run) makes a low-level
> distinction between "binary" files and "text" files. I believe it is
> because Windows guarantees (or must guarantee) certain data
> transformations on text that do not apply to other forms of data.
> (I.e. translating <cr><lf> sequences into C newline characters, etc)

AFAIK it’s not that low level. I can see no sign of it in CreateFile,
ReadFile, etc, which are approximately equivalent to open(), read() etc
in the Windows API. I think the implementation is in the C runtime
(effectively the Libc).

Cygwin does not appear rely on Windows for the implementation of its
O_TEXT and O_BINARY - you can see the translation code in
fhandler_base::read.

--
https://www.greenend.org.uk/rjk/

On Tue, 20 Feb 2024 22:25:05 +0800, Paul Edwards wrote:

> On 20/02/24 22:14, Lew Pitcher wrote:
>> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>>
>>> Hi.
>>>
>>> Cygwin has an O_TEXT on the open() call to let
>>> "the OS layer" (can be quibbled) that the file
>>> is being opened in text mode. And as such, it
>>> gives that layer an opportunity to insert CRs.
>> [snip]
>>> I'd like to add an O_TEXT flag to Linux.[snip]
>>> Any suggestions?
>>
>> Yes, one. Dont.
>>
>> Microsoft Windows (on top of which Cygwin must run) makes a low-level
>> distinction between "binary" files and "text" files. I believe it is
>> because Windows guarantees (or must guarantee) certain data
>> transformations on text that do not apply to other forms of data.
>> (I.e. translating <cr><lf> sequences into C newline characters, etc)
>>
>> HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
>> and binary data, neither in the standard I/O library, nor in the OS
>> itself.
>>
>> You are effectively asking that Linux /NOW/ make such a distinction,
>> even if only to ignore the O_TEXT flag that you ask for.
>
> The C library that most people use already has
> that distinction - and it is indeed ignored.
> Works perfectly fine.

Lets take an accounting...

1) POSIX open() does not make any distinction between "text" and "binary"
files

2) Standard C / POSIX fopen() /does/ make a distinction between "text"
and "binary" files. POSIX issues a caveat regarding the mode
parameter that "The character 'b' shall have no effect, but is allowed
for ISO C standard conformance.", referring to the mode character "b"
specified by the ISO C standard as indicating that the file should be
opened for "binary" I/O, rather than the default "text" I/O.

3) You specifically reference the open() call, not the fopen() call.

So, you ask that the open() call, a call that, under Linux and other
POSIX-compatible operating systems, makes no distinction between binary
and text files, now recognize a flag indicating that the open() should
act apon a text file.

That leaves the Linux kernel developers with a choice to make: either
a) accept your request and add an O_TEXT flag that conditions the I/O
for text data, or
b) accept your request and add an O_TEXT flag that does nothing, or
c) ignore or deny your request

Choice (a) would drastically change the behaviour of the open()
call, and any existing code that uses open() to access a text file would
have to change to specify the O_TEXT flag. I believe that this would
be a no-starter, even if the default for O_TEXT would leave behaviour
as it currently is.

Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
It would be a header change that reserves a flag that would be ignored
by everyone. I believe that the maintainers would simply veto this
as being a change that makes no change.

So, choice (c) is my bet. The maintainers simply ignore (if they want
to be polite) or deny your proposal.

>
>> It ain't gonna fly. Such a flag would invalidate /decades/ worth of
>> Unix/Linux code, and /not/ provide any functionality either to
>> the OS layer, the stdio library layer, or the application code layer.
>
> Sorry - what is being invalidated? Can you give an
> example?

Today, I can
open("/etc/passwd",O_RDONLY)
and read text from the text file /etc/passwd. I can
also
open("/sbin/init",O_RDONLY)
and read binary data from the binary file /sbin/init

If I understand correctly, you propose that code now
must
open("/etc/passwd",O_RDONLY | O_TEXT)
to read text from the text file /etc/passwd.

What would happen if, under your proposal, code still
executed
open("/etc/passwd",O_RDONLY)
? Could the code still read() text data, or would the
open() fail on something like EACCESS or EINVAL or
EOPNOTSUPP? What /would/ the consequences be of setting
O_TEXT?

Of course, if you answer that, under a POSIX-like system,
there would be /no/ consequences to O_TEXT, then you
have presented choice (b), a change that changes nothing.

--
Lew Pitcher
"In Skills We Trust"

Paul Edwards <mutazilah@gmail.com> writes:
> On 20/02/24 17:53, Richard Kettlewell wrote:
>> Paul Edwards <mutazilah@gmail.com> writes:
>>> I'd like to add an O_TEXT flag to Linux. It would be good if Linux
>>> officially reserved a flag for this use, but in the absence of that,
>>> I'm happy to create my own.
>>
>> You’d need to ask on the linux-kernel mailing list, I think. But I
>> wouldn’t predict a very positive response.
>
> That's what I'm predicting too. So can you suggest
> a flag? I think Linux is consuming flags going up,
> so I could go down for anything I need, and have
> 0x80000000 mean "O_TEXT".

No, I can’t suggest a flag. I have no more control over the Linux ABI
than you do. That’s why I pointed you to linux-kernel.

> Any suggestions within the constraints of the Linux people having
> different goals to me?

I think you should rethink your goals and/or your approach to meeting
them. open/read/write is the wrong place to translate between line
ending conventions.

--
https://www.greenend.org.uk/rjk/

On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:

> Hi.
>
> Cygwin has an O_TEXT on the open() call to let
> "the OS layer" (can be quibbled) that the file
> is being opened in text mode. And as such, it
> gives that layer an opportunity to insert CRs.
[snip]
> I am
> still missing the same thing that Cygwin was
> missing, and added O_TEXT to solve.

And, what would that be? What does the Cygwin
O_TEXT flag on open(2) do?

[snip]
> And it is not important to maintain the same value
> as Cygwin, because Cygwin creates PE executables,
> while mine is for ELF executables.

Are you just trying to keep your Linux source
source-compatible with your (Cygwin-based) Windows
source? Or are you actually looking for the functionality
behind the O_TEXT flag?

If you just want to keep source compatibility, I don't see
any problem with using a system-specific predefined macro
to implement some conditional compilation. Something like

#ifndef O_TEXT
#define O_TEXT 0
#endif

open("/etc/passwd",O_RDONLY | O_TEXT);

HTH
--
Lew Pitcher
"In Skills We Trust"

On 21/02/24 01:55, Lew Pitcher wrote:
> On Tue, 20 Feb 2024 22:25:05 +0800, Paul Edwards wrote:
>
>> On 20/02/24 22:14, Lew Pitcher wrote:
>>> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>>>
>>>> Hi.
>>>>
>>>> Cygwin has an O_TEXT on the open() call to let
>>>> "the OS layer" (can be quibbled) that the file
>>>> is being opened in text mode. And as such, it
>>>> gives that layer an opportunity to insert CRs.
>>> [snip]
>>>> I'd like to add an O_TEXT flag to Linux.[snip]
>>>> Any suggestions?
>>>
>>> Yes, one. Dont.
>>>
>>> Microsoft Windows (on top of which Cygwin must run) makes a low-level
>>> distinction between "binary" files and "text" files. I believe it is
>>> because Windows guarantees (or must guarantee) certain data
>>> transformations on text that do not apply to other forms of data.
>>> (I.e. translating <cr><lf> sequences into C newline characters, etc)
>>>
>>> HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
>>> and binary data, neither in the standard I/O library, nor in the OS
>>> itself.
>>>
>>> You are effectively asking that Linux /NOW/ make such a distinction,
>>> even if only to ignore the O_TEXT flag that you ask for.
>>
>> The C library that most people use already has
>> that distinction - and it is indeed ignored.
>> Works perfectly fine.
>
> Lets take an accounting...
>
> 1) POSIX open() does not make any distinction between "text" and "binary"
> files
>
> 2) Standard C / POSIX fopen() /does/ make a distinction between "text"
> and "binary" files. POSIX issues a caveat regarding the mode
> parameter that "The character 'b' shall have no effect, but is allowed
> for ISO C standard conformance.", referring to the mode character "b"
> specified by the ISO C standard as indicating that the file should be
> opened for "binary" I/O, rather than the default "text" I/O.
>
> 3) You specifically reference the open() call, not the fopen() call.
>
> So, you ask that the open() call, a call that, under Linux and other
> POSIX-compatible operating systems, makes no distinction between binary
> and text files, now recognize a flag indicating that the open() should
> act apon a text file.
>
> That leaves the Linux kernel developers with a choice to make: either
> a) accept your request and add an O_TEXT flag that conditions the I/O
> for text data, or
> b) accept your request and add an O_TEXT flag that does nothing, or
> c) ignore or deny your request
>
> Choice (a) would drastically change the behaviour of the open()
> call, and any existing code that uses open() to access a text file would
> have to change to specify the O_TEXT flag. I believe that this would
> be a no-starter, even if the default for O_TEXT would leave behaviour
> as it currently is.
>
> Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
> It would be a header change that reserves a flag that would be ignored
> by everyone.

Almost everyone.

PDOS/386 would use it.

> I believe that the maintainers would simply veto this
> as being a change that makes no change.
>
> So, choice (c) is my bet. The maintainers simply ignore (if they want
> to be polite) or deny your proposal.

Sure. Hence my question.

I'll take a random flag myself. Any suggestions?

>>> It ain't gonna fly. Such a flag would invalidate /decades/ worth of
>>> Unix/Linux code, and /not/ provide any functionality either to
>>> the OS layer, the stdio library layer, or the application code layer.
>>
>> Sorry - what is being invalidated? Can you give an
>> example?
>
> Today, I can
> open("/etc/passwd",O_RDONLY)
> and read text from the text file /etc/passwd. I can
> also
> open("/sbin/init",O_RDONLY)
> and read binary data from the binary file /sbin/init
>
> If I understand correctly, you propose that code now
> must
> open("/etc/passwd",O_RDONLY | O_TEXT)
> to read text from the text file /etc/passwd.
>
> What would happen if, under your proposal, code still
> executed
> open("/etc/passwd",O_RDONLY)
> ? Could the code still read() text data, or would the
> open() fail on something like EACCESS or EINVAL or
> EOPNOTSUPP? What /would/ the consequences be of setting
> O_TEXT?
>
> Of course, if you answer that, under a POSIX-like system,
> there would be /no/ consequences to O_TEXT, then you
> have presented choice (b), a change that changes nothing.

Correct. Nothing is invalidated.

It is only when run on a Windows-like system that
something happens.

BFN. Paul.

On 21/02/24 03:34, Lew Pitcher wrote:
> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>
>> Hi.
>>
>> Cygwin has an O_TEXT on the open() call to let
>> "the OS layer" (can be quibbled) that the file
>> is being opened in text mode. And as such, it
>> gives that layer an opportunity to insert CRs.
> [snip]
>> I am
>> still missing the same thing that Cygwin was
>> missing, and added O_TEXT to solve.
>
> And, what would that be? What does the Cygwin
> O_TEXT flag on open(2) do?

It will add CRs to such files on a write().

> [snip]
>> And it is not important to maintain the same value
>> as Cygwin, because Cygwin creates PE executables,
>> while mine is for ELF executables.
>
> Are you just trying to keep your Linux source
> source-compatible with your (Cygwin-based) Windows
> source? Or are you actually looking for the functionality
> behind the O_TEXT flag?

The latter. Functionality on PDOS/386, not Linux.

And not Linux source. Linux binaries.

> If you just want to keep source compatibility, I don't see
> any problem with using a system-specific predefined macro
> to implement some conditional compilation. Something like
>
> #ifndef O_TEXT
> #define O_TEXT 0
> #endif
>
> open("/etc/passwd",O_RDONLY | O_TEXT);

Not what I'm after.

BFN. Paul.

On Wed, 21 Feb 2024 05:37:45 +0800, Paul Edwards wrote:

> On 21/02/24 01:55, Lew Pitcher wrote:
>> On Tue, 20 Feb 2024 22:25:05 +0800, Paul Edwards wrote:
>>
>>> On 20/02/24 22:14, Lew Pitcher wrote:
>>>> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>>>>
>>>>> Hi.
>>>>>
>>>>> Cygwin has an O_TEXT on the open() call to let
>>>>> "the OS layer" (can be quibbled) that the file
>>>>> is being opened in text mode. And as such, it
>>>>> gives that layer an opportunity to insert CRs.
>>>> [snip]
>>>>> I'd like to add an O_TEXT flag to Linux.[snip]
>>>>> Any suggestions?
>>>>
>>>> Yes, one. Dont.
>>>>
>>>> Microsoft Windows (on top of which Cygwin must run) makes a low-level
>>>> distinction between "binary" files and "text" files. I believe it is
>>>> because Windows guarantees (or must guarantee) certain data
>>>> transformations on text that do not apply to other forms of data.
>>>> (I.e. translating <cr><lf> sequences into C newline characters, etc)
>>>>
>>>> HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
>>>> and binary data, neither in the standard I/O library, nor in the OS
>>>> itself.
>>>>
>>>> You are effectively asking that Linux /NOW/ make such a distinction,
>>>> even if only to ignore the O_TEXT flag that you ask for.
>>>
>>> The C library that most people use already has
>>> that distinction - and it is indeed ignored.
>>> Works perfectly fine.
>>
>> Lets take an accounting...
>>
>> 1) POSIX open() does not make any distinction between "text" and "binary"
>> files
>>
>> 2) Standard C / POSIX fopen() /does/ make a distinction between "text"
>> and "binary" files. POSIX issues a caveat regarding the mode
>> parameter that "The character 'b' shall have no effect, but is allowed
>> for ISO C standard conformance.", referring to the mode character "b"
>> specified by the ISO C standard as indicating that the file should be
>> opened for "binary" I/O, rather than the default "text" I/O.
>>
>> 3) You specifically reference the open() call, not the fopen() call.
>>
>> So, you ask that the open() call, a call that, under Linux and other
>> POSIX-compatible operating systems, makes no distinction between binary
>> and text files, now recognize a flag indicating that the open() should
>> act apon a text file.
>>
>> That leaves the Linux kernel developers with a choice to make: either
>> a) accept your request and add an O_TEXT flag that conditions the I/O
>> for text data, or
>> b) accept your request and add an O_TEXT flag that does nothing, or
>> c) ignore or deny your request
>>
>> Choice (a) would drastically change the behaviour of the open()
>> call, and any existing code that uses open() to access a text file would
>> have to change to specify the O_TEXT flag. I believe that this would
>> be a no-starter, even if the default for O_TEXT would leave behaviour
>> as it currently is.
>>
>> Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
>> It would be a header change that reserves a flag that would be ignored
>> by everyone.
>
> Almost everyone.
>
> PDOS/386 would use it.

If PDOS/386 needs it, then PDOS/386 probably should use it.

However, the needs of PDOS/386 aren't the needs of Linux or POSIX, and
neither Linux in particular, nor POSIX in general, need bother with
it.

If you are concerned with PDOS/386 being able to use source code written
to the POSIX standard, then I suggest that you select the value and
treatment of your O_TEXT flag to work in absence, as POSIX code won't
have it or use it.

The alternative is to specify that such a flag /is/ required for PDOS/386,
which restricts you to using code written specifically for PDOS/386, and
not the more general-purpose and widely available POSIX standard.

[snip]

Best of luck.
--
Lew Pitcher
"In Skills We Trust"

On 21/02/24 10:51, Lew Pitcher wrote:
> On Wed, 21 Feb 2024 05:37:45 +0800, Paul Edwards wrote:

>>> Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
>>> It would be a header change that reserves a flag that would be ignored
>>> by everyone.
>>
>> Almost everyone.
>>
>> PDOS/386 would use it.
>
> If PDOS/386 needs it, then PDOS/386 probably should use it.

Not so much "needs" as "would be desirable".

> However, the needs of PDOS/386 aren't the needs of Linux or POSIX, and
> neither Linux in particular, nor POSIX in general, need bother with
> it.

Sure. As I said in a previous message - "different goals".

> If you are concerned with PDOS/386 being able to use source code written
> to the POSIX standard,

Not just source - that is easy to deal with.
The executable.

> then I suggest that you select the value and
> treatment of your O_TEXT flag to work in absence, as POSIX code won't
> have it or use it.

That's exactly what I'm doing. I'm just asking
which bit would be good to use. I showed which
bit that Cygwin was using, and I made a suggestion
that I could work downwards on the assumption
that Linux is working upwards.

> The alternative is to specify that such a flag /is/ required for PDOS/386,

That's the alternative I am after.

> which restricts you to using code written specifically for PDOS/386, and

It is only code that uses the new flag that will
work nicely on PDOS/386, yes.

The other code would still run, it just won't
automatically put the CR before the NL, so it
won't look good on (this) Windows-like environment.

> not the more general-purpose

I consider adding O_TEXT to be more general-purpose
than one that assumes the code will only be used on
a Unix-like environment, never a Windows-like
environment.

> and widely available POSIX standard.

It still follows the rest of that, so it will still
work on any POSIX environment.

The *application* code will need to have:

#ifndef O_TEXT
/* uh oh, we're on a standard POSIX environment,
missing the PDOS/386-inspired Linux enhancement
that hasn't YET (as of 2024-02-21) made it to a
formal POSIX standard */
#define O_TEXT 0
#endif

.... open(... | O_TEXT ...)

BFN. Paul.

On 20/02/2024 22.40, Paul Edwards wrote:
> On 21/02/24 03:34, Lew Pitcher wrote:
>> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:

>> [snip]
>>> And it is not important to maintain the same value
>>> as Cygwin, because Cygwin creates PE executables,
>>> while mine is for ELF executables.
>>
>> Are you just trying to keep your Linux source
>> source-compatible with your (Cygwin-based) Windows
>> source? Or are you actually looking for the functionality
>> behind the O_TEXT flag?
>
> The latter. Functionality on PDOS/386, not Linux.

Think you need to look into the windows source code, as open() already
exists in windows and those cygwin don't need to reimplement that, it
only provides things that do not exists in windows but is part of POSIX.

> And not Linux source. Linux binaries.

Cygwin is not:

- a way to run native Linux apps on Windows. You must rebuild your
application from source if you want it to run on Windows.

If you want to make changes to POSIX API, you need to start contacting
Austin Group, with time changes will appear in Linux. I doubt you would
have more luck with the Austin Group than convincing Linus to add a
patch to add something that Linux don't need.

--
//Aho

Paul Edwards <mutazilah@gmail.com> writes:
> On 21/02/24 10:51, Lew Pitcher wrote:
>> If you are concerned with PDOS/386 being able to use source code written
>> to the POSIX standard,
>
> Not just source - that is easy to deal with.
> The executable.

Executables won’t have your extra flag unless you’ve personally modified
and recompiled them.

>> The alternative is to specify that such a flag /is/ required for
>> PDOS/386,
>
> That's the alternative I am after.
>
>> which restricts you to using code written specifically for PDOS/386,
>> and
>
> It is only code that uses the new flag that will work nicely on
> PDOS/386, yes.

How will you modify other syscalls (for example fstat and lseek) to be
play nicely with the newline translation?

--
https://www.greenend.org.uk/rjk/

On 21/02/24 17:48, Richard Kettlewell wrote:
> Paul Edwards <mutazilah@gmail.com> writes:
>> On 21/02/24 10:51, Lew Pitcher wrote:
>>> If you are concerned with PDOS/386 being able to use source code written
>>> to the POSIX standard,
>>
>> Not just source - that is easy to deal with.
>> The executable.
>
> Executables won’t have your extra flag unless you’ve personally modified
> and recompiled them.

Yes, that's exactly what I will do. I will start
producing executables that have that flag.

At least MY executables will live up to MY
requirements.

Which is all I'm really after.

If someone else finds that useful, for the same
reasons I find that useful, great.

If I am alone in the world, that's fine too.

The latter state could change at a later date
though. You never know what will happen a
millenia from now.

>>> which restricts you to using code written specifically for PDOS/386,
>>> and
>>
>> It is only code that uses the new flag that will work nicely on
>> PDOS/386, yes.
>
> How will you modify other syscalls (for example fstat and lseek) to be
> play nicely with the newline translation?

I only intend to run C90-compliant programs
(in this case, with a statically-linked C
runtime library).

fstat is not used by PDPCLIB, nor part of the
C90 standard.

lseek is used by PDPCLIB - that's how fseek()
is implemented.

According to C90, fseek() is not expected to
behave sensibly on text files.

And indeed - it won't (on PDOS/386 anyway; when
run on Linux it will be fine).

BFN. Paul.

On 21/02/24 15:06, J.O. Aho wrote:

>>>> And it is not important to maintain the same value
>>>> as Cygwin, because Cygwin creates PE executables,
>>>> while mine is for ELF executables.
>>>
>>> Are you just trying to keep your Linux source
>>> source-compatible with your (Cygwin-based) Windows
>>> source? Or are you actually looking for the functionality
>>> behind the O_TEXT flag?
>>
>> The latter. Functionality on PDOS/386, not Linux.
>
> Think you need to look into the windows source code, as open() already
> exists in windows

There is no such syscall. kernel32 exports CreateFile.

Regardless - Windows won't run the Linux ELF binary -
not even the one I produce myself - regardless. Although
if you install WSL it would, but that's installing
Linux - and of course Linux programs run under Linux.

It is PDOS/386 that will run it. And receive the
INT 80H. And check for the flag.

Because PDOS/386 is sort of a DOS/Windows clone,
all files are expected to use CRLF as line terminators.
That's the user requirements, basically.

So I'm trying to get my Linux ELF binaries (ones that
I build myself) to fit into that existing user
requirement.

> and those cygwin don't need to reimplement that, it
> only provides things that do not exists in windows but is part of POSIX.

It is Cygwin that invented O_TEXT, not Windows.
To satisfy that same user requirement above.

>> And not Linux source. Linux binaries.
>
> Cygwin is not:
>
> - a way to run native Linux apps on Windows. You must rebuild your
> application from source if you want it to run on Windows.

I know.

That's why I'm not using Cygwin, I'm using PDOS/386.

Which ALREADY runs a hello world Linux ELF binary.

(unlike Cygwin - which never did)

> If you want to make changes to POSIX API, you need to start contacting
> Austin Group, with time changes will appear in Linux.

https://en.wikipedia.org/wiki/Austin_Group

Ok, thanks.

> I doubt you would
> have more luck with the Austin Group than convincing Linus to add a
> patch to add something that Linux don't need.

Ok. I'll try the Austin Group first though. As you
noted, this isn't for Linux. It's for (quibbling
aside) a competitor of Linux, so the Austin Group
would be more appropriate.

BFN. Paul.

Paul Edwards <mutazilah@gmail.com> writes:
> On 21/02/24 17:48, Richard Kettlewell wrote:
>> How will you modify other syscalls (for example fstat and lseek) to be
>> play nicely with the newline translation?
>
> I only intend to run C90-compliant programs (in this case, with a
> statically-linked C runtime library).
>
> fstat is not used by PDPCLIB, nor part of the C90 standard.

Standard C doesn’t have open(), read() or write() either, but you seem
to want to modify the behaviour of those.

--
https://www.greenend.org.uk/rjk/

On 21/02/24 20:25, Richard Kettlewell wrote:
> Paul Edwards <mutazilah@gmail.com> writes:
>> On 21/02/24 17:48, Richard Kettlewell wrote:
>>> How will you modify other syscalls (for example fstat and lseek) to be
>>> play nicely with the newline translation?
>>
>> I only intend to run C90-compliant programs (in this case, with a
>> statically-linked C runtime library).
>>
>> fstat is not used by PDPCLIB, nor part of the C90 standard.
>
> Standard C doesn’t have open(), read() or write() either,

Indeed - because PDPCLIB's implementation of fopen()
needs to do the open syscall (INT 80H EAX = 5H).

fopen already knows whether the file is being
opened as text or binary, and I don't want that
information lost when I do that interrupt above.
Even though Linux (currently, and very likely
forever) won't use it, PDOS/386 has a use for it
(in the short term).

> but you seem to want to modify the behaviour of those.

I don't think that is an accurate statement.

What behavior, where?

Adding an extra flag doesn't change the behavior
of anything anywhere, immediately.

PDOS/386 would then be in a position to have a
particular behavior, but it's not really a
change, is it? It's new development/enhancement.

BFN. Paul.

On 21/02/2024 12.57, Paul Edwards wrote:
> On 21/02/24 15:06, J.O. Aho wrote:
>
>>>>> And it is not important to maintain the same value
>>>>> as Cygwin, because Cygwin creates PE executables,
>>>>> while mine is for ELF executables.
>>>>
>>>> Are you just trying to keep your Linux source
>>>> source-compatible with your (Cygwin-based) Windows
>>>> source? Or are you actually looking for the functionality
>>>> behind the O_TEXT flag?
>>>
>>> The latter. Functionality on PDOS/386, not Linux.
>>
>> Think you need to look into the windows source code, as open() already
>> exists in windows
>
> There is no such syscall. kernel32 exports CreateFile.

_sopen_s() (secure version of _open()) and _O_TEXT/_O_BINARY do exists
and I think this is what Cygwin wraps to open() and O_TEXT/O_BINARY.

> Because PDOS/386 is sort of a DOS/Windows clone,
> all files are expected to use CRLF as line terminators.
> That's the user requirements, basically.

Why don't you ask Microsoft to make ms-windows POSIX compatible, then
this issue would disappear at once ;)

> So I'm trying to get my Linux ELF binaries (ones that
> I build myself) to fit into that existing user
> requirement.

That's why other projects has solved it as define O_TEXT/O_BINARY to 0x0
if they are missing when they want to support multiple OS:es. They don't
expect that all OS has AmigaOS dos library implemented to open files
IDOS->Open("c:dir", MODE_OLDFILE)

--
//Aho

Groovy hepcat Lew Pitcher was jivin' in alt.os.linux on Wed, 21 Feb 2024
04:55 am. It's a cool scene! Dig it.

> On Tue, 20 Feb 2024 22:25:05 +0800, Paul Edwards wrote:
>> On 20/02/24 22:14, Lew Pitcher wrote:
>>> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>>>
>>>> Cygwin has an O_TEXT on the open() call to let
>>>> "the OS layer" (can be quibbled) that the file
>>>> is being opened in text mode. And as such, it
>>>> gives that layer an opportunity to insert CRs.
>>> [snip]
>>>> I'd like to add an O_TEXT flag to Linux.[snip]
>>>> Any suggestions?
>>>
>>> Yes, one. Dont.

[Snip.]

>> The C library that most people use already has
>> that distinction - and it is indeed ignored.
>> Works perfectly fine.
>
> Lets take an accounting...
>
> 1) POSIX open() does not make any distinction between "text" and
> "binary"
> files
>
> 2) Standard C / POSIX fopen() /does/ make a distinction between "text"
> and "binary" files. POSIX issues a caveat regarding the mode
> parameter that "The character 'b' shall have no effect, but is
> allowed for ISO C standard conformance.", referring to the mode
> character "b" specified by the ISO C standard as indicating that
> the file should be opened for "binary" I/O, rather than the default
> "text" I/O.
>
> 3) You specifically reference the open() call, not the fopen() call.
>
> So, you ask that the open() call, a call that, under Linux and other
> POSIX-compatible operating systems, makes no distinction between
> binary and text files, now recognize a flag indicating that the open()
> should act apon a text file.
>
> That leaves the Linux kernel developers with a choice to make: either
> a) accept your request and add an O_TEXT flag that conditions the I/O
> for text data, or
> b) accept your request and add an O_TEXT flag that does nothing, or
> c) ignore or deny your request
>
> Choice (a) would drastically change the behaviour of the open()
> call, and any existing code that uses open() to access a text file
> would have to change to specify the O_TEXT flag. I believe that this
> would be a no-starter, even if the default for O_TEXT would leave
> behaviour as it currently is.
>
> Choice (b) is ... useless. It recognizes a non-issue, and does
> nothing. It would be a header change that reserves a flag that would
> be ignored by everyone. I believe that the maintainers would simply
> veto this as being a change that makes no change.

This would be useless, true. But it would be harmless. It could be
defined simply as

#define O_TEXT 0

which would mean a mode argument of, say, O_RDONLY | O_TEXT would
ammount to O_RDONLY | 0, which would in turn ammount to simply
O_RDONLY.
I can't see the kernel devs going for it, though.

> So, choice (c) is my bet. The maintainers simply ignore (if they want
> to be polite) or deny your proposal.
>>
>>> It ain't gonna fly. Such a flag would invalidate /decades/ worth of
>>> Unix/Linux code, and /not/ provide any functionality either to
>>> the OS layer, the stdio library layer, or the application code
>>> layer.
>>
>> Sorry - what is being invalidated? Can you give an
>> example?
>
> Today, I can
> open("/etc/passwd",O_RDONLY)
> and read text from the text file /etc/passwd. I can
> also
> open("/sbin/init",O_RDONLY)
> and read binary data from the binary file /sbin/init
>
> If I understand correctly, you propose that code now
> must
> open("/etc/passwd",O_RDONLY | O_TEXT)
> to read text from the text file /etc/passwd.
>
> What would happen if, under your proposal, code still
> executed
> open("/etc/passwd",O_RDONLY)
> ? Could the code still read() text data, or would the
> open() fail on something like EACCESS or EINVAL or
> EOPNOTSUPP? What /would/ the consequences be of setting
> O_TEXT?

I think what the OP is proposing is option (b) above: allow an O_TEXT
identifier having no effect.

> Of course, if you answer that, under a POSIX-like system,
> there would be /no/ consequences to O_TEXT, then you
> have presented choice (b), a change that changes nothing.

Right. This is trivially doable. But, as I said, I can't see this
happening.
To the OP: a solution to your problem is simple. Put the following
lines in your source code before any use of O_TEXT:

#ifndef O_TEXT
#define O_TEXT 0
#endif

--

----- Dig the NEW and IMPROVED news sig!! -----

-------------- Shaggy was here! ---------------
Ain't I'm a dawg!!

On 21/02/24 11:36, Peter 'Shaggy' Haywood wrote:

>> b) accept your request and add an O_TEXT flag that does nothing, or
>>
>> Choice (b) is ... useless. It recognizes a non-issue, and does
>> nothing. It would be a header change that reserves a flag that would
>> be ignored by everyone. I believe that the maintainers would simply
>> veto this as being a change that makes no change.
>
> This would be useless, true. But it would be harmless. It could be
> defined simply as
>
> #define O_TEXT 0

No. It can't be that. It needs to be non-zero so
that I can detect it in a non-Linux environment.

> I think what the OP is proposing is option (b) above: allow an O_TEXT
> identifier having no effect.

No effect ... on Linux.

An effect ... on PDOS/386.

> To the OP: a solution to your problem is simple. Put the following
> lines in your source code before any use of O_TEXT:
>
> #ifndef O_TEXT
> #define O_TEXT 0
> #endif

No, that doesn't solve my problem.

When the ELF executable is run on PDOS/386, I will
have no way of knowing that this open is text, and
thus PDOS/386 (the OS) should add CRs whenever it
sees a LF.

Because we're beyond the point when anyone else
would add the CR.

BFN. Paul.

Paul Edwards <mutazilah@gmail.com> writes:
> On 21/02/24 20:25, Richard Kettlewell wrote:
>> Standard C doesn’t have open(), read() or write() either,
>
> Indeed - because PDPCLIB's implementation of fopen()
> needs to do the open syscall (INT 80H EAX = 5H).
>
> fopen already knows whether the file is being opened as text or
> binary, and I don't want that information lost when I do that
> interrupt above. Even though Linux (currently, and very likely
> forever) won't use it, PDOS/386 has a use for it (in the short term).
>
>> but you seem to want to modify the behaviour of those.
>
> I don't think that is an accurate statement.
>
> What behavior, where?

You’ve been talking about doing translation _somewhere_, and cited
Cygwin as an example in your first posting, and have been consistently
using the flag name from Cygwin. Cygwin’s O_TEXT causes read() and
write() to translate between newline conventions, so if you meant
something else then it was a confusing decision to use Cygwin as an
example.

If, in fact, you don’t want to add translation to read() and write()
then you don’t need to add any new flags to open(). As an existing
example, Windows manages to do translation in its <stdio.h>
implementation without any assistance from CreateFile and its
friends. Presumably it records the text/binary distinction in its
implementation of the FILE type.

--
https://www.greenend.org.uk/rjk/

"J.O. Aho" <user@example.net> writes:
> _sopen_s() (secure version of _open()) and _O_TEXT/_O_BINARY do exists
> and I think this is what Cygwin wraps to open() and O_TEXT/O_BINARY.

Cygwin it has its own independent implementation of newline translation.

--
https://www.greenend.org.uk/rjk/

On 21/02/24 20:52, J.O. Aho wrote:

>>> Think you need to look into the windows source code, as open() already
>>> exists in windows
>>
>> There is no such syscall. kernel32 exports CreateFile.
>
> _sopen_s() (secure version of _open()) and _O_TEXT/_O_BINARY do exists
> and I think this is what Cygwin wraps to open() and O_TEXT/O_BINARY.

Thanks for the pointer.

That allowed me to find this:

https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/open-wopen?view=msvc-170

https://learn.microsoft.com/en-us/cpp/c-runtime-library/text-and-binary-mode-file-i-o?view=msvc-170

And those functions exist here:

C:\WINNT\system32>odwin -x msvcrt.dll | grep -i open
[ 218] _fdopen
[ 242] _fsopen
[ 403] _open
[ 404] _open_osfhandle
[ 414] _popen
[ 445] _sopen
[ 527] _wfdopen
[ 534] _wfopen
[ 535] _wfreopen
[ 536] _wfsopen
[ 547] _wopen
[ 550] _wpopen
[ 558] _wsopen
[ 620] fopen
[ 628] freopen

C:\WINNT\system32>

(on my Windows 2000 system)

However, do note that this msvcrt.dll is undocumented
and nominally not meant to be used directly, and only
as part of Visual Studio (and even then, it's no longer
called that - it gets a version number added).

So this is very different from a kernel32.dll equivalent
of a syscall.

However, it does at least give prior art - on top of
Cygwin.

So it is something that I can potentially go to the
Austin Group with.

>> Because PDOS/386 is sort of a DOS/Windows clone,
>> all files are expected to use CRLF as line terminators.
>> That's the user requirements, basically.
>
> Why don't you ask Microsoft to make ms-windows POSIX compatible, then
> this issue would disappear at once ;)

Well that's what everyone has been saying for decades
now, and Microsoft wins every time.

So you can continue fighting that war - no problem.

However, I do want to approach it from the other
direction - a Windows clone.

And I've been working on mine for decades too.

>> So I'm trying to get my Linux ELF binaries (ones that
>> I build myself) to fit into that existing user
>> requirement.
>
> That's why other projects has solved it as define O_TEXT/O_BINARY to 0x0
> if they are missing when they want to support multiple OS:es.

That solves a different problem a different way.

> They don't
> expect that all OS has AmigaOS dos library implemented to open files
> IDOS->Open("c:dir", MODE_OLDFILE)

Well that's an interesting analogy.

One that I have already visited.

So, let's start with PCs with the same processor,
in this case 68000. Atari is a competitor to the
Amiga. Can the Atari run Amiga software?

The equivalent of "everyone should be POSIX" is
"everyone should be Atari". And that's what Atari
thinks. And the Amiga will win every time.

So, my approach - can the Atari run Amiga software?
At least C90-compliant software.

And the answer is - no. At their heart, the Amiga
executables work (without DLLs) by hardcoding the
number 4. All the Amigados functions hang off that.

And on the Atari, address 4 cannot be used.

So should we give up?

No.

First of all, we change the Amiga applications - in
a way that doesn't affect them running on the Amiga.

See here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/amistart.asm

I carry an alternate sysbase in d7. It won't be set
when running under genuine AmigaOS, but when run on
a competitor - perhaps not Atari's OS, but some
other OS running on Atari hardware - like, PDOS/Atari
or something like that - then d7 is set to a register
that points to memory that the Atari can actually
address properly.

And then those - select - AmigaOS executables - basically
ones compiled against PDPCLIB which is the only C library
I know of that implements the d7 protocol - which I only
created a couple of years ago - will run on alternate
hardware/OS.

And now I'm doing the same with Linux executables.

First I change the Linux executables with a view to
getting them to run on competing vendor platforms,
build those executables, and then later bring the
competing platform up to speed.

The d7 Amiga "standard" has been invented years (so
far) in advance of a competitor actually arising.

But that's for a different reason - I'm not that
interested in the Amiga API.

But I am interested in the OS/2 2.0 API.

And indeed - I have already prepared - or in the
midst of preparing - modifications there too.
Instead of depending on the 16-bit thunk libraries
to exist, I'm getting raw keyboard input by using
DosDevIOCtl. You can see that here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/stdio.c#l2184

So now I am getting into position for PDOS/386 to
support the 32-bit MSDOS API (which I sort of
created myself because Microsoft didn't do that),
Win32, OS/2 2.0 and Linux ELF.

All in the same OS, all with very little code to
support the different OSes, and the OS hopefully
fits on a 360k floppy even with that extra
functionality (or should come very close, anyway -
noting that it isn't a design goal to fit on a
360k floppy - 720k would be period accurate I think).

BFN. Paul.