On Mon, 22 Jan 2024 18:43:06 GMT, Scott Lurndal wrote:

> AIX had this odd 'overcommit' feature, which allowed the system to
> overcommit memory resources. It would allocate virtual address space
> (via malloc) even if there wasn't enough memory + backing store to
> support it, and would take a SIGSEGV when referenced if, at the time of
> reference, there was still insufficient memory and/or backing store to
> support allocating a physical page to that portion of the VA space.
>
> Linux can be configured to operate in the same way.

Not quite the same. Linux overcommit means “never having to say no”. If
the system runs short of memory, then this wakes up the Dreaded OOM
Killer, which starts scoring processes on their hunger for memory, and
killing the ones that are the biggest memory hogs.

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

> Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
>
>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>>
>>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

[...]

>>>> I think I would have *liked* to see C23 drop the special permission
>>>> to return a null pointer for a requested size of zero. C11 says (and
>>>> this applies to malloc, realloc, and all other allocation functions):
>>>> [...]
>>>>
>>>> Note that malloc(0) or realloc(ptr, 0) can still fail and return
>>>> a null pointer if no space can be allocated, so all allocations
>>>> should still be checked. But with this proposed change, code
>>>> could rely on realloc(ptr, 0) returning a non-null pointer *unless*
>>>> available memory is critically low -- pretty much the same as in C11,
>>>> except that a null pointer would be an indication that something
>>>> is seriously wrong.
>>>>
>>>> (I remember seeing a discussion about making the behavior of
>>>> realloc(ptr, 0) undefined. I'm making inquiries, and I'll follow
>>>> up if I learn anything relevant.)
>>>
>>> I got a response from JeanHeyd Meneide.
>>>
>>> If realloc(ptr, 0) returns a null pointer there's no way to tell
>>> whether allocation failed (and ptr has not been freed), or the
>>> implementation returns a null pointer for zero-sized allocations
>>> (and ptr has been freed). Some implementations set errno, but C
>>> doesn't require it.

(Let me mention in passing that there is a way to tell that
should work in practice on any non-malicious implementation.)

>> It's trivial to fix that problem: simply require implementations
>> to define a preprocessor symbol about how the implementation
>> works. Problem solved.
>>
>> (There are other instances of implementation-defined behavior
>> that would benefit from analogous changes along these lines.)
>
> I tend to agree that such a preprocessor symbol would be an
> improvement.
>
> I still think, as I wrote above, that removing the permission to
> return a null pointer on a successful zero-sized allocation would
> be a greater improvement.
>
> A preprocessor symbol makes it easier for programmers to work
> around the potential difference between implementations. The
> change I advocate would make it completely unnecessary.
>
> Except, of course, that most code would still have to allow for
> pre-C26 behavior, even if the change were adopted in C26. That's
> unavoidable in the absence of time machines. On the gripping
> hand, since it seems that most existing implementations (well, all
> of the few I've tried) return a non-null pointer for malloc(0), it
> might be reasonable to ignore the few pre-C26 implementations that
> return a null pointer.

I have an observation worth mentioning. Using glibc (tested in
Ubuntu), these assignments

void *x = malloc( 0 );
void *y = malloc( 1 );

both return a non-null pointer. Following that, however, these
calls

void *xx = realloc( x, 0 );
void *yy = realloc( y, 0 );

both return a null pointer, and subsequent code shows that both x
and y have indeed been deallocated. (Both gcc and clang give the
same result, which isn't surprising since both are using glibc.)

On Mon, 22 Jan 2024 16:34:55 GMT, Scott Lurndal wrote:

> I see no problem in special casing [zero].

Special-casing zero is something I thought we had left behind in the bad
old days of FORTRAN IV.

James Kuyper <jameskuyper@alumni.caltech.edu> writes:

> On 1/22/24 18:12, Blue-Maned_Hawk wrote:
>
>> Kaz Kylheku wrote:
>>
>>> On 2024-01-19, Blue-Maned_Hawk <bluemanedhawk@invalid.invalid> wrote:
>>>
>>>> Kaz Kylheku wrote:
>>>>
>>>>> Of course, that's not what "behavior can be readily inferred"
>>>>> means, no matter where I got the example.
>>>>
>>>> Inferences may be makeable, but they have no power.
>>>
>>> Yeah, tell that to your compiler when it surprisingly deletes
>>> code based on inference.
>>
>> Apologies, let me clarify: In a technical standard, inferences
>> from nonnormative text cannot be used to make normative statements.
>
> True, but the relevant inferences were from normative text. The
> context of the above comments has been snipped, so I'll repeat it
> here:
>
> On 1/17/24 19:10, Kaz Kylheku wrote:
>
>> On 2024-01-17, Blue-Maned_Hawk <bluemanedhawk@invalid.invalid> wrote:
>>
>>> It looks to me like it was always undefined behavior, but it
>>> just wasn't explicitly stated as such in C99.
>>
>> That is incorrect. A behavior for the following can be readily
>> inferred from C99:
>>
>> void *p = malloc(42);
>
> Per C99:
> "The malloc function returns either a null pointer or a pointer to
> the allocated space." (7.22.3.4p2)
>
> Thus, there are two possibilities:
> A: p == NULL
> B: p contains the address of a block of memory at least 42 bytes
> long, suitably aligned to store an object of any type.
>
> The standard does not mandate documentation of which of the two
> applies, so this has unspecified behavior.

The return value from malloc() is not unspecified behavior. In
n1256, see section 7.20.3 paragraph 1, and also the definition of
unspecified behavior in 3.4.4.

On Tue, 23 Jan 2024 19:28:38 -0800, Tim Rentsch wrote:

> ... however, these calls
>
> void *xx = realloc( x, 0 );
> void *yy = realloc( y, 0 );
>
> both return a null pointer, and subsequent code shows that both x and y
> have indeed been deallocated. (Both gcc and clang give the same result,
> which isn't surprising since both are using glibc.)

You could have just read the docs
<https://manpages.debian.org/3/realloc.en.html>.

But remember, there is no requirement for gcc, at least, to use glibc. You
can build programs with it that link to alternatives C runtimes.

On 2024-01-24, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> I have an observation worth mentioning. Using glibc (tested in
> Ubuntu), these assignments
>
> void *x = malloc( 0 );
> void *y = malloc( 1 );
>
> both return a non-null pointer. Following that, however, these
> calls
>
> void *xx = realloc( x, 0 );
> void *yy = realloc( y, 0 );
>
> both return a null pointer, and subsequent code shows that both x
> and y have indeed been deallocated. (Both gcc and clang give the
> same result, which isn't surprising since both are using glibc.)

The behavior is documented by Glibc.

https://www.gnu.org/software/libc/manual/html_node/Changing-Block-Size.html

"If you pass a null pointer for ptr, realloc behaves just like ‘malloc
(newsize)’. Otherwise, if newsize is zero realloc frees the block and
returns NULL."

(It'a also documented sloppily by that hapless source of misinformation
known as the Linux man pages project, which says that realloc(ptr, 0) is
equivalent to free(ptr). neglecting to make any remarks on the return
value.

For a good laugh, check out the inane diatribes in "man 7 regex".)

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 2024-01-24, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
> You could have just read the docs
><https://manpages.debian.org/3/realloc.en.html>.

You're mistaken; those are not the docs.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 24/01/2024 04:28, Tim Rentsch wrote:
> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>
>> Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
>>
>>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

>>>> I got a response from JeanHeyd Meneide.
>>>>
>>>> If realloc(ptr, 0) returns a null pointer there's no way to tell
>>>> whether allocation failed (and ptr has not been freed), or the
>>>> implementation returns a null pointer for zero-sized allocations
>>>> (and ptr has been freed). Some implementations set errno, but C
>>>> doesn't require it.
>
> (Let me mention in passing that there is a way to tell that
> should work in practice on any non-malicious implementation.)
>

No, let's not let you mention that in passing. It will lead to another
endless thread where people ask for an explanation of your method, while
you pop by for a few days a month to tell us what you mean by
"non-malicious".

I cannot be alone in being curious as to what this method is, and when
it might or might not be applicable.

Please either explain yourself, or stop making such claims.

(Note to others - Tim either does not read my posts, or actively refuses
to reply to them. If anyone else wants to know his method, they will
need to reply themselves.)

Lawrence D'Oliveiro <ldo@nz.invalid> writes:
>On Mon, 22 Jan 2024 18:43:06 GMT, Scott Lurndal wrote:
>
>> AIX had this odd 'overcommit' feature, which allowed the system to
>> overcommit memory resources. It would allocate virtual address space
>> (via malloc) even if there wasn't enough memory + backing store to
>> support it, and would take a SIGSEGV when referenced if, at the time of
>> reference, there was still insufficient memory and/or backing store to
>> support allocating a physical page to that portion of the VA space.
>>
>> Linux can be configured to operate in the same way.
>
>Not quite the same. Linux overcommit means “never having to say no”. If
>the system runs short of memory, then this wakes up the Dreaded OOM
>Killer, which starts scoring processes on their hunger for memory, and
>killing the ones that are the biggest memory hogs.

The behavior of overcommit is configurable in linux. That's just one
possibility.

David Brown <david.brown@hesbynett.no> writes:
> On 24/01/2024 04:28, Tim Rentsch wrote:
>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>>> Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
>>>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>>>>> I got a response from JeanHeyd Meneide.
>>>>>
>>>>> If realloc(ptr, 0) returns a null pointer there's no way to tell
>>>>> whether allocation failed (and ptr has not been freed), or the
>>>>> implementation returns a null pointer for zero-sized allocations
>>>>> (and ptr has been freed). Some implementations set errno, but C
>>>>> doesn't require it.

To avoid any confusion, I wrote the above paragraph; JeanHeyd Meneide
(the editor for C23 standard) did not.

>> (Let me mention in passing that there is a way to tell that
>> should work in practice on any non-malicious implementation.)
>
> No, let's not let you mention that in passing. It will lead to
> another endless thread where people ask for an explanation of your
> method, while you pop by for a few days a month to tell us what you
> mean by "non-malicious".
>
> I cannot be alone in being curious as to what this method is, and when
> it might or might not be applicable.
>
> Please either explain yourself, or stop making such claims.
>
> (Note to others - Tim either does not read my posts, or actively
> refuses to reply to them. If anyone else wants to know his method,
> they will need to reply themselves.)

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 1/24/24 04:41, David Brown wrote:
> On 24/01/2024 04:28, Tim Rentsch wrote:
....
>> (Let me mention in passing that there is a way to tell that
>> should work in practice on any non-malicious implementation.)
....
> Please either explain yourself, or stop making such claims.

As we all know, Tim will do neither of those things.

> (Note to others - Tim either does not read my posts, or actively
> refuses to reply to them. If anyone else wants to know his method,
> they will need to reply themselves.)

He still seems to be reading at least some of mine.

Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
> Richard Kettlewell <invalid@invalid.invalid> writes:
>> Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
>>> Richard Kettlewell <invalid@invalid.invalid> writes:
>>>
>>>> Of course that's what the cautious programmer must do practice. But in
>>>> terms of the total cost (over all users, implementers, etc) fixing the
>>>> definitions of memcpy/memset/etc (as well as malloc) would have been the
>>>> better answer.
>>>
>>> Better in some ways, worse in others. Better for me is not
>>> always the same as better for thee.
>>
>> The way I think it's better is total cost over all users, implementers,
>> etc. [...]
>
> Yes, I got that. Not everyone shares that view.

The cost is an objective statement, not a view.

In principle I could be wrong that the cost of everyone dealing with the
resulting bugs, adding workarounds, etc, is greater than the cost of
fixing implementations, but frankly it doesn’t seem very plausible.

> It's not even a given that everyone thinks that's the best metric to
> optimize.

C’s history since standardization does include a disregard for the costs
born by its users, yes.

>>>> Standard C is trying to have its own cake and eat it here: 0-sized
>>>> allocations can be represented by null pointers when it's malloc, but
>>>> not when it's memcpy.
>>>
>>> Actually the two decisions have essentially nothing to do with
>>> each other. You might want to read what the C Rationale
>>> document has to say about the decisions behind various
>>> memory allocation policies.
>>
>> Do you mean
>> https://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf
>> s7.20.3?
>
> I was looking at C99RationaleV5.10, although I believe the comments
> are from the earlier, original Rationale document. I expect you
> have the right section; I don't have the document open to check.

It is thoroughly unenlightening as an explanation.

* It says the rules about zero-length allocations are intended to
permit a particular paradigm, of which an example is given.

* The example includes no attempt at error handling, which is where the
pain point from the permission for malloc(0)=NULL arises. So it seems
like the author of that section didn’t consider the impact of the
rules adopted.

* Nothing about the example depends either on malloc(0)=NULL or
malloc(0)!=NULL, making it mysterious why the example is said to
support the decisions made in C89.

* The text talks about the distinction betwen “nothing” and “zero” but
in the context of malloc the distinction the caller actually needs to
make is between success and failure.

* The principle that “zero-length objects” are invalid does not seem to
be rationalized anywhere in the text. While there may be some
advantage to it, nobody wrote them down.

A couple of other notes:

* The Unix V7 rules for malloc neatly bypass any concerns about
zero-length objects (if anybody cared about that at the time); the
definition is “returns a pointer to a block of at least size bytes”.
malloc(0) returning a pointer to a 1-byte block would be a non-issue
under this definition regardless of any theoretical objections.

* If it happens that c=0 at any point in the loop, the new C23 rules
make the example in 7.20.3 undefined behavior. I guess the example has
fallen out of favour.

Finally back to the relationship to malloc:

> Actually the two decisions have essentially nothing to do with
> each other.

It does seem like nobody considered the question of whether the two
decisions were compatible; or at least if they did consider it then
their reasoning was never added to the rationale.

--
https://www.greenend.org.uk/rjk/

Kaz Kylheku <433-929-6894@kylheku.com> writes:

> On 2024-01-21, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> Kaz Kylheku <433-929-6894@kylheku.com> writes:
>>
>> [.. considering the behavior of malloc(0) ..]
>>
>>> [If] implentations could have a single dedicated object for
>>> representing empty allocations (which can be passed to free any
>>> number of times), that would also be a nice requirement.
>>
>> That defeats the whole purpose of having malloc(0) return
>> a non-null value. Don't you understand anything?
>
> I undertand very clearly from past disccussions that you're attached
> to a particular use case whereby malloc(0) returns unique objects.

What I like or don't like has nothing to do with it. The motivating
principle comes from the group that originally advocated it (one of
whom was Doug McIlroy, IIRC), long before there ever was a C
standard.

> However, I don't see that as the purpose, let alone the "whole
> purpose".

Then I suggest you read up more on the history of the early
discussions.

> The standard currently does not endorse malloc(0) as a factory
> for unique pointers.

The C standard requires that all non-null return values from the
memory allocation functions be distinct: "Each such allocation
shall yield a pointer to an object disjoint from any other object."
See, eg, section 7.20.3, paragraph 1, in n1256.

> Currently, it cannot be relied on due to the possibility of the null
> return. If you want that behavior portably, you currently have to
> use malloc(1) instead, or some other nonzero value. Even if your
> program detects that malloc(0) returns a non-null pointer one time,
> there is no requirement that all subsequent such allocations will
> return non-null.

Your understanding of the term "implementation-defined" is flawed.
Each implementation must make a particular choice and stick to it.
In cases where the choice is allowed to vary the C standard uses
different language, as for example "in an implementation-defined
manner". Note also the item in Annex J.3.12: "Whether the calloc,
malloc, and realloc functions return a null pointer or a pointer to
an allocated object when the size requested is zero". Not "when" or
"under what circumstances" but "whether". Granted, text in Annex J
is non-normative, but surely it reflects the meaning intended by the
C standard's authors.

On 2024-01-24 01:15:19 Tim Rentsch wrote:
> James Kuyper <james...@alumni.caltech.edu> writes:
>
>> On 1/22/24 18:12, Blue-Maned_Hawk wrote:
....
>>> void *p = malloc(42);
>>
>> Per C99:
>> "The malloc function returns either a null pointer or a pointer to
>> the allocated space." (7.22.3.4p2)
>>
>> Thus, there are two possibilities:
>> A: p == NULL
>> B: p contains the address of a block of memory at least 42 bytes
>> long, suitably aligned to store an object of any type.
>>
>> The standard does not mandate documentation of which of the two
>> applies, so this has unspecified behavior.
>
> The return value from malloc() is not unspecified behavior. In
> n1256, see section 7.20.3 paragraph 1, and also the definition of
> unspecified behavior in 3.4.4.

3.4.4p1:
"unspecified behavior
use of an unspecified value, or other behavior where this International
Standard provides two or more possibilities and imposes no further
requirements on which is chosen in any instance"

7.20.3p1:
"The order and contiguity of storage allocated by successive calls to
the calloc, malloc, and realloc functions is unspecified. The pointer
returned if the allocation succeeds is suitably aligned so that it may
be assigned to a pointer to any type of object and then used to access
such an object or an array of such objects in the space allocated (until
the space is explicitly deallocated). The lifetime of an allocated
object extends from the allocation until the deallocation. Each such
allocation shall yield a pointer to an object disjoint from any other
object. The pointer returned points to the start (lowest byte address)
of the allocated space. If the space cannot be allocated, a null pointer
is returned. If the size of the space requested is zero, the behavior is
implementation- defined: either a null pointer is returned, or the
behavior is as if the size were some nonzero value, except that the
returned pointer shall not be used to access an object."

Agreed - the value returned from malloc() is not unspecified behavior. I
didn't say that it was. It is specified precisely which value must be
returned, depending upon whether or not malloc() could allocate the
space. What is unspecified by the standard is whether or not the space
can be allocated. The standard provides two possibilities, allocable or
not allocable, and imposes no requirements on which is chosen in any
instance. That matches the definition of unspecified behavior.

If you think that this is an inappropriate application of that term,
that is irrelevant to the point I was making. What is relevant is that
there are two possible outcomes, and that neither of them involves
undefined behavior, and that neither of them sets up a situation that
gives the call to realloc() undefined behavior.

On 2024-01-25, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> Your understanding of the term "implementation-defined" is flawed.

No it isn't.

Obviously, an implementation cannot vaccilate among different choices of
how malloc(0) should behave.

Just like an implementation cannot say that, in even-numbered lines of
code, the right shift of a negative integer will propagate the sign,
wheras in odd lines it will propagate a zero.

Rather, the issue is that null is always a valid return, regardless
of the choice of behavior.

If an implementation is observed as returning non-null from malloc(0),
then we know that its choice for implementation-defined behavior is to
do that.

But then if it returns null from another call to malloc(0), we
cannot conclude that its nonconforming, because that null could
be an indication of allocation failure, which is always allowed.

We /can/ conclude that the allocation failed. But so what?

We cannot write a run-time test for the behavior that is guaranteed to
terminate.

bool malloc_0_non_null = false;

for (;;) {
void *ptr = malloc(0);

if (ptr) {
malloc_0_non_null = true;
free(ptr);
break;
}
}

This will obviously not terminate on an implementation that always
returns null; so we need to somehow cut the loop short. For instance by
capping the number of iterations.

Problem is, we don't know the minimum number of iterations any given
implementation will require. I.e. how many initial allocation failures
the implementation will report before changing its mind and handing out
the non-null.

With the malloc zero ambiguity, ISO C had successfully added a new
formally undecidable problem to the field of computer science
(cue slow clap).

I apologize for not making all this reasoning clearer in my posting.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Kaz Kylheku <433-929-6894@kylheku.com> writes:
[...]
> If an implementation is observed as returning non-null from malloc(0),
> then we know that its choice for implementation-defined behavior is to
> do that.
>
> But then if it returns null from another call to malloc(0), we
> cannot conclude that its nonconforming, because that null could
> be an indication of allocation failure, which is always allowed.
>
> We /can/ conclude that the allocation failed. But so what?
[...]

If malloc(0) consistently returns a null pointer and malloc(1)
consistently return a non-null pointer *and* the implementations's
documentation says that a successful malloc(0) returns a non-null
pointer, we can conclude with high confidence that the implementation is
non-conforming. (It could be made conforming by changing the
documentation.)

Or that "the behavior is as if the size were some nonzero value", and
the nonzero value chosen by the implementation is SIZE_MAX. That would
be deliberately perverse. It's probably what the DeathStation 9000
does.

Or that, just by coincidence, those malloc(0) calls happened to fail;
thus "high confidence" rather than certainty.

Unfortunately, code does not have access to the implementation's
documentation.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 1/25/2024 11:35 AM, Keith Thompson wrote:
[...]
> Unfortunately, code does not have access to the implementation's
> documentation.

Mind if I quote that from time to time? Nice one.

"Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> writes:
> On 1/25/2024 11:35 AM, Keith Thompson wrote:
> [...]
>> Unfortunately, code does not have access to the implementation's
>> documentation.
>
> Mind if I quote that from time to time? Nice one.

Feel free.

To expand on that, I think it was Tim Rentsch who recently suggested
adding predefined macros that code can query to find out which choices
an implementation has made. I'm concerned that that would introduce a
lot of new symbols into the global namespace, so I'm thinking about a
new preprocessor pseudo-function, something like:

#if IMPL(MALLOC0, NON_NULL_RESULT)
...
#elif IMPL(MALLOC0, NULL_RESULT)
...
#endif

This is just off the top of my head. On the other hand, a bunch of
macros that can be queried with #ifdef would be simpler to implement.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

On 25/01/2024 22:40, Keith Thompson wrote:
> "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> writes:
>> On 1/25/2024 11:35 AM, Keith Thompson wrote:
>> [...]
>>> Unfortunately, code does not have access to the implementation's
>>> documentation.
>>
>> Mind if I quote that from time to time? Nice one.
>
> Feel free.
>
> To expand on that, I think it was Tim Rentsch who recently suggested
> adding predefined macros that code can query to find out which choices
> an implementation has made. I'm concerned that that would introduce a
> lot of new symbols into the global namespace, so I'm thinking about a
> new preprocessor pseudo-function, something like:
>
> #if IMPL(MALLOC0, NON_NULL_RESULT)
> ...
> #elif IMPL(MALLOC0, NULL_RESULT)
> ...
> #endif
>
> This is just off the top of my head. On the other hand, a bunch of
> macros that can be queried with #ifdef would be simpler to implement.
>

I believe you can have your cake and eat it here.

IMPL could be implemented as :

#define IMPL(__group, __detail) defined(_IMPLEMENTATION_DETAIL_ ## \
__group ## _IS_ ## __detail)

Then the implementation detail would just have to define macros such as:

#define _IMPLEMENTATION_DETAIL_MALLOC0_IS_NON_NULL_RESULT

That would be easy to implement and avoid a risk of identifier collisions.

All you need to do now is persuade the main C compiler and library
implementers to follow this new standard !

Kaz Kylheku <433-929-6894@kylheku.com> writes:

> On 2024-01-23, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> Richard Kettlewell <invalid@invalid.invalid> writes:
>>
>>> Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
>>>
>>>> Richard Kettlewell <invalid@invalid.invalid> writes:
>>>>
>>>>> Of course that's what the cautious programmer must do practice. But in
>>>>> terms of the total cost (over all users, implementers, etc) fixing the
>>>>> definitions of memcpy/memset/etc (as well as malloc) would have been the
>>>>> better answer.
>>>>
>>>> Better in some ways, worse in others. Better for me is not
>>>> always the same as better for thee.
>>>
>>> The way I think it's better is total cost over all users, implementers,
>>> etc. [...]
>>
>> Yes, I got that. Not everyone shares that view. It's not
>> even a given that everyone thinks that's the best metric
>> to optimize.
>
> It is an inherently collectivist view; I would expect communists
> to agree with it unanimously. (But then bungle the execution by
> forming a five year plan in which the work is divided into groups
> driven by meaningless local metrics and irrelevant incentives.)

All science trembles at the searing logic of your fiery intellect.

Richard Kettlewell <invalid@invalid.invalid> writes:

> Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
>
>> Richard Kettlewell <invalid@invalid.invalid> writes:
>>
>>> Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
>>>
>>>> Richard Kettlewell <invalid@invalid.invalid> writes:
>>>>
>>>>> Of course that's what the cautious programmer must do practice.
>>>>> But in terms of the total cost (over all users, implementers,
>>>>> etc) fixing the definitions of memcpy/memset/etc (as well as
>>>>> malloc) would have been the better answer.
>>>>
>>>> Better in some ways, worse in others. Better for me is not
>>>> always the same as better for thee.
>>>
>>> The way I think it's better is total cost over all users,
>>> implementers, etc. [...]
>>
>> Yes, I got that. Not everyone shares that view.
>
> The cost is an objective statement, not a view.

What I mean is that not everyone agrees with the idea that
changing the definitions of memcpy/memset/etc will achieve the
goal of minimizing the total cost of ownership.

> In principle I could be wrong that the cost of everyone dealing
> with the resulting bugs, adding workarounds, etc, is greater than
> the cost of fixing implementations, but frankly it doesn't seem
> very plausible.

My comment is only about what other people consider to be so,
not to assess the question itself.

>> It's not even a given that everyone thinks that's the best metric
>> to optimize.
>
> C's history since standardization does include a disregard for the
> costs born by its users, yes.

I don't think that's true. Probably it is true that the
priorities of folks who have worked on the ISO C committee
don't exactly match your priorities.

>>>>> Standard C is trying to have its own cake and eat it here:
>>>>> 0-sized allocations can be represented by null pointers when
>>>>> it's malloc, but not when it's memcpy.
>>>>
>>>> Actually the two decisions have essentially nothing to do with
>>>> each other. You might want to read what the C Rationale
>>>> document has to say about the decisions behind various
>>>> memory allocation policies.
>>>
>>> Do you mean
>>> https://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf
>>> s7.20.3?
>>
>> I was looking at C99RationaleV5.10, although I believe the comments
>> are from the earlier, original Rationale document. I expect you
>> have the right section; I don't have the document open to check.
>
> It is thoroughly unenlightening as an explanation.

Oh. I suppose that depends on what it is one thinks the writing
is trying to explain.

> * It says the rules about zero-length allocations are intended to
> permit a particular paradigm, of which an example is given.

Actually it doesn't say that. What it does say is that the
treatment of such cases was guided in part by a desire to
support the given pattern.

> * The example includes no attempt at error handling, which is
> where the pain point from the permission for malloc(0)=NULL
> arises. So it seems like the author of that section didn't
> consider the impact of the rules adopted.

The code pattern shown is something that was observed to be in
widespread use, not something that the Rationale's authors
constructed to illustrate a point. It isn't surprising that
there isn't any mention of error handling, because it isn't
important to what the section is trying to explain.

> * Nothing about the example depends either on malloc(0)=NULL or
> malloc(0)!=NULL, making it mysterious why the example is said
> to support the decisions made in C89.

Nothing in the Rationale document says that the given code
pattern supports the decisions made in the C standard. Any
implication actually goes the other direction, that decisions
about how malloc() works were guided (in part) by a desire to
support the given code pattern.

> * The text talks about the distinction betwen 'nothing' and 'zero'
> but in the context of malloc the distinction the caller actually
> needs to make is between success and failure.

That may indeed be an important distinction, but it is not
important to what this part of the Rationale document is trying
to explain.

> * The principle that 'zero-length objects' are invalid does not
> seem to be rationalized anywhere in the text. While there may
> be some advantage to it, nobody wrote them down.

What the text actually says is that the C89 Committee decided not
to accept the idea of zero-length objects. Probably the phrasing
could have been better; in any case what is meant is that they
decided not to /require/ support of zero-length objects, but
rather have it be optional (and implementation-defined).

> A couple of other notes:
>
> * The Unix V7 rules for malloc neatly bypass any concerns about
> zero-length objects (if anybody cared about that at the time);
> the definition is "returns a pointer to a block of at least size
> bytes". malloc(0) returning a pointer to a 1-byte block would be
> a non-issue under this definition regardless of any theoretical
> objections.

I don't know anything about what Unix V7 says about malloc(), so
I have nothing to say about that.

> * If it happens that c=0 at any point in the loop, the new C23
> rules make the example in 7.20.3 undefined behavior. I guess
> the example has fallen out of favour.

Invalidating working code is a significant part of why people
object to the decision to make zero-sized realloc() calls be
undefined behavior.

> Finally back to the relationship to malloc:
>
>> Actually the two decisions have essentially nothing to do with
>> each other.
>
> It does seem like nobody considered the question of whether the
> two decisions were compatible; [...]

They may or may not have. You may want to read the Introduction
section of the Rationale document.

Kaz Kylheku <433-929-6894@kylheku.com> writes:

> On 2024-01-25, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> Your understanding of the term "implementation-defined" is flawed.
>
> No it isn't.
>
> Obviously, an implementation cannot vaccilate among different
> choices of how malloc(0) should behave. [...]

I think you've lost track of what point it is you are trying
to make. I know I have.

> I apologize for not making all this reasoning clearer in my
> posting.

I recommend adopting the habit of re-reading (and revising) what
you have written, before posting it. I try to do this at least
once before posting, and often two or three times. A technique
that can be very effective is speaking the text out loud as you
read it.

A piece of advice that appears in most of the essays I have read
about writing is something like, The essence of good writing is
re-writing. Another one that I like (this one from Fred Brooks)
is, Easy writing makes hard reading, and vice versa. Perhaps
these suggestions will help you avoid confusing writing in the
future.

Tim Rentsch <tr.17687@z991.linuxsc.com> writes:
> Richard Kettlewell <invalid@invalid.invalid> writes:
>> C's history since standardization does include a disregard for the
>> costs born by its users, yes.
>
> I don't think that's true. Probably it is true that the
> priorities of folks who have worked on the ISO C committee
> don't exactly match your priorities.

The practical impacts and the complaints have been widespread for many
years. It’s not just me.

>> It is thoroughly unenlightening as an explanation.
>
> Oh. I suppose that depends on what it is one thinks the writing
> is trying to explain.

You were the one who recommend it in the context of this thread...

It claims to be trying to explain the treatment of null pointers and
zero-length allocations. You may have to take it on trust, but for
someone who happens not to like the treatment chosen, it singularly
fails to explain that.

>> * The example includes no attempt at error handling, which is
>> where the pain point from the permission for malloc(0)=NULL
>> arises. So it seems like the author of that section didn't
>> consider the impact of the rules adopted.
>
> The code pattern shown is something that was observed to be in
> widespread use, not something that the Rationale's authors
> constructed to illustrate a point. It isn't surprising that
> there isn't any mention of error handling, because it isn't
> important to what the section is trying to explain.

The implication is that error handling (and its relevance to reliability
and robustness) was not of interest.

>> * Nothing about the example depends either on malloc(0)=NULL or
>> malloc(0)!=NULL, making it mysterious why the example is said
>> to support the decisions made in C89.
>
> Nothing in the Rationale document says that the given code
> pattern supports the decisions made in the C standard. Any
> implication actually goes the other direction, that decisions
> about how malloc() works were guided (in part) by a desire to
> support the given code pattern.

The point is alternative decisions (for example, the V7 behavior) would
have supported it just as well.

>> * The principle that 'zero-length objects' are invalid does not
>> seem to be rationalized anywhere in the text. While there may
>> be some advantage to it, nobody wrote them down.
>
> What the text actually says is that the C89 Committee decided not
> to accept the idea of zero-length objects. Probably the phrasing
> could have been better; in any case what is meant is that they
> decided not to /require/ support of zero-length objects, but
> rather have it be optional (and implementation-defined).

The text describes requiring zero-length objects as a “compelling
theoretical disadvantage”, without explaining why it’s so compelling or
even why requiring them would be a disadvantage. Earlier implementations
apparently didn’t find any difficulty with them, or at least
insufficient difficulty to influence the rules for malloc.

>> A couple of other notes:
>>
>> * The Unix V7 rules for malloc neatly bypass any concerns about
>> zero-length objects (if anybody cared about that at the time);
>> the definition is "returns a pointer to a block of at least size
>> bytes". malloc(0) returning a pointer to a 1-byte block would be
>> a non-issue under this definition regardless of any theoretical
>> objections.
>
> I don't know anything about what Unix V7 says about malloc(), so
> I have nothing to say about that.

You know what I told you above.

>> * If it happens that c=0 at any point in the loop, the new C23
>> rules make the example in 7.20.3 undefined behavior. I guess
>> the example has fallen out of favour.
>
> Invalidating working code is a significant part of why people
> object to the decision to make zero-sized realloc() calls be
> undefined behavior.

The rationale notes that the C89 rules change the behavior of existing
code (despite “existing code is important, existing implementations are
not” and “avoid quiet changes”).

>> Finally back to the relationship to malloc:
>>
>>> Actually the two decisions have essentially nothing to do with
>>> each other.
>>
>> It does seem like nobody considered the question of whether the
>> two decisions were compatible; [...]
>
> They may or may not have. You may want to read the Introduction
> section of the Rationale document.

Being more specific about what you’re getting at might cause me to want
to read that in more detail. So far I’ve just identified a couple of
“important principles” that were violated by the rules C89 introduced
for malloc.

--
https://www.greenend.org.uk/rjk/

On 2024-01-26, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> Kaz Kylheku <433-929-6894@kylheku.com> writes:
>
>> On 2024-01-25, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>>
>>> Your understanding of the term "implementation-defined" is flawed.
>>
>> No it isn't.
>>
>> Obviously, an implementation cannot vaccilate among different
>> choices of how malloc(0) should behave. [...]
>
> I think you've lost track of what point it is you are trying
> to make. I know I have.

Here I'm simply defending myself against the accusation that I don't
understand the definition of implementation-defined behavior. (Based on
what I'm guessing might be the basis of the perceived misunderstanding.)

This is a mostly self-contained topic, which has a connection to an
earlier point, that point being a relatively minor.

I walked into it like you wanted me to, executing your plan
to disconnect.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On 1/25/24 13:49, Kaz Kylheku wrote:
> On 2024-01-25, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>> Your understanding of the term "implementation-defined" is flawed.
>
> No it isn't.
>
> Obviously, an implementation cannot vaccilate among different choices of
> how malloc(0) should behave.

That might seem reasonable, but the committee has ruled otherwise. The
committee's official response to a rather early DR (unfortunately, I
cannot remember which one, and I can't figure out how to search for it -
therefore, you're free to disbelieve what I'm about to say) said that
when something is unspecified, an implementation is free to make
different choices in different programs; or in different translation
units in the same program, or on different lines of the same translation
unit, or in different sub-expressions of the same expression, or in
different runs of the same program, or in different executions of the
same expression.

This obviously cannot be true in general. Most importantly, many aspects
of the representations of types are unspecified, but it would make it
impossible for different programs to communicate with each other through
a file, or even for different parts of a program to communicate with
each other through the values stored in variables, if the
representations were allowed to change. Since I've been unable to locate
the DR, I cannot verify what the committee said about that issue if
anything. However, in any case where it's not too difficult to deal with
the possibility of an unspecified choice changing, you should generally
assume that it IS possible.

I don't think it would be particularly difficult to cope with the
possibility that malloc(0) was documented as choosing randomly whether
to return a null pointer or the same result as malloc(1).

The fact that some unspecified behavior is implementation-defined
doesn't change this - it just means that it must be documented. There's
no limit on how bizarre the way is that an implementation decides among
the choices - so long as that way is correctly documented.