In article <86y1imbopb.fsf@linuxsc.com>,
Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>cross@spitfire.i.gajendra.net (Dan Cross) writes:
>> In article <86a5vq1s8y.fsf@linuxsc.com>,
>> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>>
>>> cross@spitfire.i.gajendra.net (Dan Cross) writes:
>>>
>>>> In article <86v8i54yk5.fsf@linuxsc.com>,
>>>> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>>>>
>>>>> I think you misunderstood my comment. I didn't mean to say
>>>>> anything about what the authors say about the evolution of
>>>>> realloc(). I meant only that the code they gave works just fine
>>>>> (not counting the problem when 'sizeof (int) == 1' that Ben B
>>>>> pointed out) -- importantly, under the assumptions that the
>>>>> authors explicitly stated -- and that they didn't say anything
>>>>> about whether the code is portable or well-defined if considered
>>>>> outside of those assumptions. The authors may have some opinions
>>>>> about whether the code /should/ work in general, but they don't
>>>>> make any claims about whether it /does/ work in general, and that
>>>>> point is the only one I mean to address.
>>>>
>>>> No, I understood that, but I disagree: I believe that they were
>>>> trying to make a more general statement, beyond simply the
>>>> circumstances they outlined.
>>>
>>> Then I think that either you are misreading what they say
>>> or you are seeing something that isn't there. I went back
>>> and thoroughly reviewed the paper, carefully reading each
>>> sentence. In no case did any statement fall outside the
>>> bounds I outlined above.
>>
>> Maybe I am; truthfully, I haven't given it (the article) much
>> thought since it came up months ago now.
>>
>> Let's look at the relevant section of the article (thankfully it
>> has been edited since initially published to remove some of the
>> more offensive language: the slur about "neurodivergent ideas"
>> is gone), with some commentary about where I'm coming from
>> interspersed:
>
>I appreciate you making an effort to write a thoughtful reply. I
>will try to respond in kind.

Certainly. But I confess that I find it very hard to have this
conversation (or any, really) with spans of weeks or months
between posts.

>> |The C89 and C99 standards committees strongly recommended that
>> |allocation interfaces malloc, calloc, and realloc return a null
>> |pointer in response to zero-byte requests.3,6 This implies that
>> |realloc(p,0) should unconditionally free(p) and return NULL: No
>> |new allocation happens in this case, so there's no possibility
>> |of an allocation failure. For brevity, let "zero-null" denote
>> |allocator implementations that comply with the C89/C99
>> |guidance.
>>
>> Ok. Here the author suggests that the C89 and C99 standards
>> suggest that `malloc(0)` really ought to return NULL.
>
>Note that he doesn't say the C89/C99 standards but the standards
>committees. The comments he's talking about are given in the C
>Rationale documents (and so indicated by the footnote numbers),
>which of course were written by committee members. You might
>want to take a look at those documents, or at least the later
>one, C99 Rationale V5 (dated April 2003); there are five or six
>paragraphs talking about this question.

I did when this discucussion first came up, but I don't see that
being particularly relevant here. In particular, it's not clear
that they're referring to the rationale, and if they were, they
could have simply stated that.

>> |The Swiss-Army-knife aspect of realloc is daunting at first,
>> |but this interface rewards patient study. Soon you realize that
>> |zero-null realloc was thoughtfully designed to enable elegant
>> |dynamic arrays that do exactly the right thing under all
>> |circumstances, obviating the need for clunky and error-prone
>> |code to handle grow-from-zero and shrink-to-zero as special
>> |cases.
>>
>> This seems specious, at best. Doug McIlroy implemented realloc
>> and this is the opposite of what he requested when it was
>> standardized:
>> https://www.tuhs.org/pipermail/tuhs/2015-September/007452.html
>>
>> Note that the original author of `realloc` requested that
>> `realloc(0)` retun something other than NULL. That directly
>> contradicts the author's suggestion that `realloc` was
>> "thoughtfully designed to enable elegant dynamic arrays".
>
>The paper does not say that. What it does say (with my emphasis
>added) is that "_zero-null_ realloc was thoughtfully designed" to
>have the mentioned property, which I believe it was.

I see no basis to believe that. Of course you may choose to,
and there's certainly nothing wrong with that, but I find it
hard to believe that it was anything other than an in-situ
interpretation made by an implementor somewhere.

Thsi is similar to how filenames starting with a '.' are
ommitted from the default output of the Unix `ls` command not
because of a conscious design decision, but because of a bug
that was simply codified over time.

A fact I have learned is that very often systems just evolve
with no clear defining principles behind that evolution. For
example, the group at Bell Labs has acknowledged that much of
what makes up the "standard C library" is simply the set of
functions that were found useful by the maintainers of 7th
Edition Unix.

>Personally
>I find dynamic arrays done using zero-null realloc semantics to
>be cleaner than those using McIlroy-style semantics (and also
>cleaner than those that have to work with both). The paper does
>not make any statement that I can see about the design process
>of zero-nonnull realloc; clearly that version of realloc is
>considered worse, in the paper's view, than zero-null realloc,
>but it doesn't say anything about how that version was designed.
>
>> Moreover, the `realloc0` function that was discussed here does
>> not strike me as either "clunky" or "error-prone", and makes the
>> authors' code well-defined on any implementation. Given their
>> noxious code style, it's litearlly a one-line fix. Sure, I
>> suppose it's subjective?
>
>You are offering a different comparison than what he is talking
>about.

Am I? Absent the author chiming in here, that's speculation.

>There is no question that having to write an extra
>function (whether realloc0 or a similar one) takes more effort
>and is more likely to cause error than simply layering on top of
>a zero-null realloc (which always does a free and returns null
>for a requested size of zero). The sentence you quote is talking
>about the zero-null style of realloc, and nothing more than that.

Then let's modify their code without a new function.

static int resize(const size_t nu) {
int *t = NULL;
if (nu > SIZE_MAX / sizeof *S) return TOOBIG;
if (nu && ((t = realloc(S, nu * sizeof *S)) == NULL) return ALLOCFAIL;
S = t;
N = nu; return 0;
}

Again, it's subjective (and honestly, that style, copied from
the authors, is beyond hideous...), but this really isn't much
different from the original, and doesn't involve an extra
function at all and once again, sidesteps the entire problem.

>> |Figure 1 illustrates idiomatic realloc via a simple stack that
>> |grows with every push() and shrinks with every pop(). Pointer S
>> |and counter N (lines 1 and 2) represent the stack: S points to
>> |an array of N strictly positive ints. Because they are
>> |statically allocated, initially the pointer is NULL and the
>> |counter is zero, indicating an empty stack. Function resize
>> |(lines 4-10) resizes the stack to a given new capacity,
>> |checking for arithmetic overflow (line 6) before calling
>> |realloc and checking the return value for memory exhaustion
>> |(line 8). Allocation failure is inferred when a nonzero new
>> |size is requested but NULL is returned; zero-null realloc also
>> |returns NULL when the second argument is zero, but this does
>> |not indicate an allocation failure because no allocation was
>> |attempted. (Checking errno doesn't enable portable code to
>> |detect allocation failure because the C standards don't say
>> |how out-of-memory affects errno.) Thanks to zero-null realloc's
>> |versatility, the resize function need not consider whether the
>> |stack is growing from zero or shrinking to zero or re-sizing in
>> |some other way; everything Just Works regardless.
>>
>> This paragraph is the root of my objection; particularly the
>> first sentence. This code is described as "idiomatic", but
>> according to whom?
>
>I believe you are misunderstanding the author's intent. He means
>the usage of realloc() is idiomatic, not that the example code is
>idiomatic.

cross@spitfire.i.gajendra.net (Dan Cross) writes:

> In article <86a5vq1s8y.fsf@linuxsc.com>,
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> cross@spitfire.i.gajendra.net (Dan Cross) writes:
>>
>>> In article <86v8i54yk5.fsf@linuxsc.com>,
>>> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>>>
>>>> I think you misunderstood my comment. I didn't mean to say
>>>> anything about what the authors say about the evolution of
>>>> realloc(). I meant only that the code they gave works just fine
>>>> (not counting the problem when 'sizeof (int) == 1' that Ben B
>>>> pointed out) -- importantly, under the assumptions that the
>>>> authors explicitly stated -- and that they didn't say anything
>>>> about whether the code is portable or well-defined if considered
>>>> outside of those assumptions. The authors may have some opinions
>>>> about whether the code /should/ work in general, but they don't
>>>> make any claims about whether it /does/ work in general, and that
>>>> point is the only one I mean to address.
>>>
>>> No, I understood that, but I disagree: I believe that they were
>>> trying to make a more general statement, beyond simply the
>>> circumstances they outlined.
>>
>> Then I think that either you are misreading what they say
>> or you are seeing something that isn't there. I went back
>> and thoroughly reviewed the paper, carefully reading each
>> sentence. In no case did any statement fall outside the
>> bounds I outlined above.
>
> Maybe I am; truthfully, I haven't given it (the article) much
> thought since it came up months ago now.
>
> Let's look at the relevant section of the article (thankfully it
> has been edited since initially published to remove some of the
> more offensive language: the slur about "neurodivergent ideas"
> is gone), with some commentary about where I'm coming from
> interspersed:

I appreciate you making an effort to write a thoughtful reply. I
will try to respond in kind.

> |The C89 and C99 standards committees strongly recommended that
> |allocation interfaces malloc, calloc, and realloc return a null
> |pointer in response to zero-byte requests.3,6 This implies that
> |realloc(p,0) should unconditionally free(p) and return NULL: No
> |new allocation happens in this case, so there's no possibility
> |of an allocation failure. For brevity, let "zero-null" denote
> |allocator implementations that comply with the C89/C99
> |guidance.
>
> Ok. Here the author suggests that the C89 and C99 standards
> suggest that `malloc(0)` really ought to return NULL.

Note that he doesn't say the C89/C99 standards but the standards
committees. The comments he's talking about are given in the C
Rationale documents (and so indicated by the footnote numbers),
which of course were written by committee members. You might
want to take a look at those documents, or at least the later
one, C99 Rationale V5 (dated April 2003); there are five or six
paragraphs talking about this question.

> |The Swiss-Army-knife aspect of realloc is daunting at first,
> |but this interface rewards patient study. Soon you realize that
> |zero-null realloc was thoughtfully designed to enable elegant
> |dynamic arrays that do exactly the right thing under all
> |circumstances, obviating the need for clunky and error-prone
> |code to handle grow-from-zero and shrink-to-zero as special
> |cases.
>
> This seems specious, at best. Doug McIlroy implemented realloc
> and this is the opposite of what he requested when it was
> standardized:
> https://www.tuhs.org/pipermail/tuhs/2015-September/007452.html
>
> Note that the original author of `realloc` requested that
> `realloc(0)` retun something other than NULL. That directly
> contradicts the author's suggestion that `realloc` was
> "thoughtfully designed to enable elegant dynamic arrays".

The paper does not say that. What it does say (with my emphasis
added) is that "_zero-null_ realloc was thoughtfully designed" to
have the mentioned property, which I believe it was. Personally
I find dynamic arrays done using zero-null realloc semantics to
be cleaner than those using McIlroy-style semantics (and also
cleaner than those that have to work with both). The paper does
not make any statement that I can see about the design process
of zero-nonnull realloc; clearly that version of realloc is
considered worse, in the paper's view, than zero-null realloc,
but it doesn't say anything about how that version was designed.

> Moreover, the `realloc0` function that was discussed here does
> not strike me as either "clunky" or "error-prone", and makes the
> authors' code well-defined on any implementation. Given their
> noxious code style, it's litearlly a one-line fix. Sure, I
> suppose it's subjective?

You are offering a different comparison than what he is talking
about. There is no question that having to write an extra
function (whether realloc0 or a similar one) takes more effort
and is more likely to cause error than simply layering on top of
a zero-null realloc (which always does a free and returns null
for a requested size of zero). The sentence you quote is talking
about the zero-null style of realloc, and nothing more than that.

> |Figure 1 illustrates idiomatic realloc via a simple stack that
> |grows with every push() and shrinks with every pop(). Pointer S
> |and counter N (lines 1 and 2) represent the stack: S points to
> |an array of N strictly positive ints. Because they are
> |statically allocated, initially the pointer is NULL and the
> |counter is zero, indicating an empty stack. Function resize
> |(lines 4-10) resizes the stack to a given new capacity,
> |checking for arithmetic overflow (line 6) before calling
> |realloc and checking the return value for memory exhaustion
> |(line 8). Allocation failure is inferred when a nonzero new
> |size is requested but NULL is returned; zero-null realloc also
> |returns NULL when the second argument is zero, but this does
> |not indicate an allocation failure because no allocation was
> |attempted. (Checking errno doesn't enable portable code to
> |detect allocation failure because the C standards don't say
> |how out-of-memory affects errno.) Thanks to zero-null realloc's
> |versatility, the resize function need not consider whether the
> |stack is growing from zero or shrinking to zero or re-sizing in
> |some other way; everything Just Works regardless.
>
> This paragraph is the root of my objection; particularly the
> first sentence. This code is described as "idiomatic", but
> according to whom?

I believe you are misunderstanding the author's intent. He means
the usage of realloc() is idiomatic, not that the example code is
idiomatic. And I think that calling the way realloc() is used
here idiomatic is a fair statement; certainly I expect there
would be no problem finding plenty of examples in existing code
that employ a similar usage.

> Moreover, that sentence doesn't explicitly
> mention "zero-null realloc"; they defer discussion of that
> until later in the paragraph (if it were me, I'd have prefaced
> the entire thing with something like, "For implementations that
> provide zero-null semantics, ...").

I think this suggested re-write changes the meaning. Clearly the
paragraph is making a point about zero-null semantics, but the
code is not meant to be limited to using zero-null realloc.

> Critically, they _don't
> describe what happens outside of their zero-null semantic
> model_. Would replacing the `realloc` call in their code with
> the ternary expression discussed earlier in this thread not be
> "idiomatic"? Could an argument be made that it is even more
> idiomatic to use an `if` statement here to guard against the
> `realloc(0)` case?

The code is there to help make a point a zero-null realloc, so I
think it's fair that it doesn't talk about what happens under a
McIlroy-style realloc. However, the code is written so as to be
usable under both kinds of realloc, so let's look at what the
behavior is under the various C standards. The code always
works without any problems on systems that have a zero-null
malloc and realloc. On systems that have a McIlroy-style
allocator:

Under C89/C90 rules, the code always works without any problems.

Under C99 and C11 rules, the code always works, except that it
can have an undiagnosed memory leak (on some but not all of those
systems) if a resize to zero cannot allocate a new object.

Under C17 rules, there is basically an implementation-defined
choice between C89/C90 rules and C99/C11 rules. So if the choice
is one way the code always works, and if the choice is the other
way an undiagnosed memory leak is possible (depending on the
particular implementation-defined choices made).

cross@spitfire.i.gajendra.net (Dan Cross) writes:

I have read through your comments. There are just a few points
where I have a response, plus at the end a more overall reaction.

> [...] I confess that I find it very hard to have this
> conversation (or any, really) with spans of weeks or months
> between posts.

Yes, I'm really sorry about that. I'm trying to do better.

> Except that the zero-length allocation _did_ cause production
> failures, [...]
>
> See https://twitter.com/__phantomderp/status/1643674954750197760
> and
> https://awakened1712.github.io/hacking/hacking-whatsapp-gif-rce/

AFAICT the twitter link is just a reference to the second URL.

The paper at the second link is not concerned with what we are
discussing; it doesn't mention using realloc(). The problem
does concern "re-allocation" but the paper says this:

Re-allocation is a combination of free and malloc. If the
size of the re-allocation is 0, it is simply a free.

Any questions about realloc() don't come into the picture here.

> Then let's modify their code without a new function.
>
> static int resize(const size_t nu) {
> int *t = NULL;
> if (nu > SIZE_MAX / sizeof *S) return TOOBIG;
> if (nu && ((t = realloc(S, nu * sizeof *S)) == NULL) return ALLOCFAIL;
> S = t;
> N = nu; return 0;
> }
>
> Again, it's subjective (and honestly, that style, copied from
> the authors, is beyond hideous...), but this really isn't much
> different from the original, and doesn't involve an extra
> function at all and once again, sidesteps the entire problem.

(The line with the complicated if() has an extra left parenthesis
but that is easily remedied.)

This code leaks memory, for reasons having nothing to do with
using realloc(). Ironically it illustrates a point given in the
Kelly/Pan paper about zero-null realloc().

> [...]

Reading through your comments was disappointing. In your earlier
posting I had the impression that you were largely reacting to the
tone of the original paper. Here it seems like you were still
doing that even while you were responding to my remarks. The
tone of my remarks is not at all like that of the paper.

Your comments haven't given me any reason to think the points in
the Kelly/Pan paper are off base. If anything, just the opposite.

In article <86r0nmwzn2.fsf@linuxsc.com>,
Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>cross@spitfire.i.gajendra.net (Dan Cross) writes:
>
>I have read through your comments. There are just a few points
>where I have a response, plus at the end a more overall reaction.
>
>> [...] I confess that I find it very hard to have this
>> conversation (or any, really) with spans of weeks or months
>> between posts.
>
>Yes, I'm really sorry about that. I'm trying to do better.

I'm sorry, but it's been nearly two weeks and this was almost
missed under a deluge of spam. Following this discussion is
simply too difficult given the amounts of time between your
responses.

Tim, I have a lot of respect for you, I really do (perhaps you
don't recall but we've corresponded several times), but without
more timely interaction this conversation is impossible for me
to maintain. I appreciate that you are trying to do better, but
it's just too awkward and honestly slightly irksome that you
would respond after so much time has elapsed.

>> Except that the zero-length allocation _did_ cause production
>> failures, [...]
>>
>> See https://twitter.com/__phantomderp/status/1643674954750197760
>> and
>> https://awakened1712.github.io/hacking/hacking-whatsapp-gif-rce/
>
>AFAICT the twitter link is just a reference to the second URL.

The twitter thread is from the co-chair of the C23 committee
when asked about the realloc(..., 0)-is-UB change.

>The paper at the second link is not concerned with what we are
>discussing; it doesn't mention using realloc(). The problem
>does concern "re-allocation" but the paper says this:
>
> Re-allocation is a combination of free and malloc. If the
> size of the re-allocation is 0, it is simply a free.
>
>Any questions about realloc() don't come into the picture here.

I think the committee cochair, and the person who wrote the
paper suggesting that realloc(..., 0), become UB, would find
that surprising as that was the example they cited when asked
about this issue. (Note that JeanHyde and Seacord both weighed
in on that Twitter thread. Of course, Twitter seems to be
broken at the moment, so I find it hard to see.

>> Then let's modify their code without a new function.
>>
>> static int resize(const size_t nu) {
>> int *t = NULL;
>> if (nu > SIZE_MAX / sizeof *S) return TOOBIG;
>> if (nu && ((t = realloc(S, nu * sizeof *S)) == NULL) return ALLOCFAIL;
>> S = t;
>> N = nu; return 0;
>> }
>>
>> Again, it's subjective (and honestly, that style, copied from
>> the authors, is beyond hideous...), but this really isn't much
>> different from the original, and doesn't involve an extra
>> function at all and once again, sidesteps the entire problem.
>
>(The line with the complicated if() has an extra left parenthesis
>but that is easily remedied.)
>
>This code leaks memory, for reasons having nothing to do with
>using realloc().

Indeed it does. Easily remedied.

// Why is this style so ugly?
static int resize(const size_t nu) {
int *t = NULL;
if (nu > SIZE_MAX / sizeof *S) return TOOBIG;
if (nu) { if ((t = realloc(S, nu * sizeof *S)) == NULL) return ALLOCFAIL; }
else free(S);
S = t;
N = nu; return 0;
}

>Ironically it illustrates a point given in the
>Kelly/Pan paper about zero-null realloc().

The Queue article potentially leaked memory (if a zero-length
allocation failed), which is a pattern that has led to
production failures, so I respectfully disagree.

I'd suggest it says more about trying to respond quickly in an
informal manner, and while mantaining this hideous style.

If it were me, I'd probably write this as:

static int
resize(const size_t nu)
{ int *t;

if (nu > SIZE_MAX / sizeof(*S))
return TOOBIG;
if (nu == 0) {
free(S);
S = NULL;
N = 0;
return 0;
}

t = realloc(S, nu * sizeof(*S));
if (t == NULL)
return ALLOCFAIL;
S = t;
N = nu;

return 0;
}

Handling the exceptional cases explicitly up front, a la
Dijkstra's guarded clauses.

>> [...]
>
>Reading through your comments was disappointing. In your earlier
>posting I had the impression that you were largely reacting to the
>tone of the original paper. Here it seems like you were still
>doing that even while you were responding to my remarks. The
>tone of my remarks is not at all like that of the paper.

I'm sorry if you feel that I was responding to your tone (which
I agree was totally unlike the Queue article); that was not my
intention, so please accept my apologies if that's how it came
across.

>Your comments haven't given me any reason to think the points in
>the Kelly/Pan paper are off base. If anything, just the opposite.

The point I have tried repeatedly to make here is that
reasonable readers can come away with different interpretations
of the Queue article.

You evidently have an interpretation that you have come to
regarding its points and validity. While I maintain that much
of that is based on speculation, I'm really not trying to
change your mind: merely explain the reasoning behind _my_
interpretation.

I do maintain that the tone and some of the content of the
Queue article is objectively offensive (particularly before the
slur about neurodivergence was removed), and I would advise
caution before we disparage those on the committee as being
either uninformed or unwise. Consider that they may be privy
to a lot of things that we, here, are not.

- Dan C.

On Mon, 28 Aug 2023 23:42:28 -0000 (UTC)
cross@spitfire.i.gajendra.net (Dan Cross) wrote:
> In article <86r0nmwzn2.fsf@linuxsc.com>,
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> >cross@spitfire.i.gajendra.net (Dan Cross) writes:
> >> See https://twitter.com/__phantomderp/status/1643674954750197760
> >> and
> >> https://awakened1712.github.io/hacking/hacking-whatsapp-gif-rce/
> >
> >AFAICT the twitter link is just a reference to the second URL.
>
> The twitter thread is from the co-chair of the C23 committee
> when asked about the realloc(..., 0)-is-UB change.
>
> >The paper at the second link is not concerned with what we are
> >discussing; it doesn't mention using realloc(). The problem
> >does concern "re-allocation" but the paper says this:
> >
> > Re-allocation is a combination of free and malloc. If the
> > size of the re-allocation is 0, it is simply a free.
> >
> >Any questions about realloc() don't come into the picture here.
>
> I think the committee cochair, and the person who wrote the
> paper suggesting that realloc(..., 0), become UB, would find
> that surprising as that was the example they cited when asked
> about this issue. (Note that JeanHyde and Seacord both weighed
> in on that Twitter thread. Of course, Twitter seems to be
> broken at the moment, so I find it hard to see.

So members of the C standard committee are discussing issues of the standard
on twitter ? Now *that's* scary.

In article <Y6Oy4M3MEjvyiR0tr@bongo-ra.co>,
Spiros Bousbouras <spibou@gmail.com> wrote:
>On Mon, 28 Aug 2023 23:42:28 -0000 (UTC)
>cross@spitfire.i.gajendra.net (Dan Cross) wrote:
>> In article <86r0nmwzn2.fsf@linuxsc.com>,
>> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>> >cross@spitfire.i.gajendra.net (Dan Cross) writes:
>> >> See https://twitter.com/__phantomderp/status/1643674954750197760
>> >> and
>> >> https://awakened1712.github.io/hacking/hacking-whatsapp-gif-rce/
>> >
>> >AFAICT the twitter link is just a reference to the second URL.
>>
>> The twitter thread is from the co-chair of the C23 committee
>> when asked about the realloc(..., 0)-is-UB change.
>>
>> >The paper at the second link is not concerned with what we are
>> >discussing; it doesn't mention using realloc(). The problem
>> >does concern "re-allocation" but the paper says this:
>> >
>> > Re-allocation is a combination of free and malloc. If the
>> > size of the re-allocation is 0, it is simply a free.
>> >
>> >Any questions about realloc() don't come into the picture here.
>>
>> I think the committee cochair, and the person who wrote the
>> paper suggesting that realloc(..., 0), become UB, would find
>> that surprising as that was the example they cited when asked
>> about this issue. (Note that JeanHyde and Seacord both weighed
>> in on that Twitter thread. Of course, Twitter seems to be
>> broken at the moment, so I find it hard to see.
>
>So members of the C standard committee are discussing issues of the standard
>on twitter ? Now *that's* scary.

More like responding to questions about this particular issue,
mostly because it was easy for me to contact JeanHyde there.

- Dan C.

cross@spitfire.i.gajendra.net (Dan Cross) writes:

> In article <86r0nmwzn2.fsf@linuxsc.com>,
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> cross@spitfire.i.gajendra.net (Dan Cross) writes:
>>
>> I have read through your comments. There are just a few points
>> where I have a response, plus at the end a more overall reaction.
>>
>>> [...] I confess that I find it very hard to have this
>>> conversation (or any, really) with spans of weeks or months
>>> between posts.
>>
>> Yes, I'm really sorry about that. I'm trying to do better.
>
> I'm sorry, but it's been nearly two weeks and this was almost
> missed under a deluge of spam. Following this discussion is
> simply too difficult given the amounts of time between your
> responses.
>
> Tim, I have a lot of respect for you, I really do (perhaps you
> don't recall but we've corresponded several times), but without
> more timely interaction this conversation is impossible for me
> to maintain.

Okay. Thank you for your comments.

cross@spitfire.i.gajendra.net (Dan Cross) writes:

> In article <86v8i54yk5.fsf@linuxsc.com>,
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> I think you misunderstood my comment. I didn't mean to say
>> anything about what the authors say about the evolution of
>> realloc(). I meant only that the code they gave works just fine
>> (not counting the problem when 'sizeof (int) == 1' that Ben B
>> pointed out) -- importantly, under the assumptions that the
>> authors explicitly stated -- and that they didn't say anything
>> about whether the code is portable or well-defined if considered
>> outside of those assumptions. The authors may have some opinions
>> about whether the code /should/ work in general, but they don't
>> make any claims about whether it /does/ work in general, and that
>> point is the only one I mean to address.
>
> No, I understood that, but I disagree: I believe that they were
> trying to make a more general statement, beyond simply the
> circumstances they outlined.

Then I think that either you are misreading what they say
or you are seeing something that isn't there. I went back
and thoroughly reviewed the paper, carefully reading each
sentence. In no case did any statement fall outside the
bounds I outlined above.

In article <86a5vq1s8y.fsf@linuxsc.com>,
Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>cross@spitfire.i.gajendra.net (Dan Cross) writes:
>
>> In article <86v8i54yk5.fsf@linuxsc.com>,
>> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>>
>>> I think you misunderstood my comment. I didn't mean to say
>>> anything about what the authors say about the evolution of
>>> realloc(). I meant only that the code they gave works just fine
>>> (not counting the problem when 'sizeof (int) == 1' that Ben B
>>> pointed out) -- importantly, under the assumptions that the
>>> authors explicitly stated -- and that they didn't say anything
>>> about whether the code is portable or well-defined if considered
>>> outside of those assumptions. The authors may have some opinions
>>> about whether the code /should/ work in general, but they don't
>>> make any claims about whether it /does/ work in general, and that
>>> point is the only one I mean to address.
>>
>> No, I understood that, but I disagree: I believe that they were
>> trying to make a more general statement, beyond simply the
>> circumstances they outlined.
>
>Then I think that either you are misreading what they say
>or you are seeing something that isn't there. I went back
>and thoroughly reviewed the paper, carefully reading each
>sentence. In no case did any statement fall outside the
>bounds I outlined above.

Maybe I am; truthfully, I haven't given it (the article) much
thought since it came up months ago now.

Let's look at the relevant section of the article (thankfully it
has been edited since initially published to remove some of the
more offensive language: the slur about "neurodivergent ideas"
is gone), with some commentary about where I'm coming from
interspersed:

|The C89 and C99 standards committees strongly recommended that
|allocation interfaces malloc, calloc, and realloc return a null
|pointer in response to zero-byte requests.3,6 This implies that
|realloc(p,0) should unconditionally free(p) and return NULL: No
|new allocation happens in this case, so there's no possibility
|of an allocation failure. For brevity, let "zero-null" denote
|allocator implementations that comply with the C89/C99
|guidance.

Ok. Here the author suggests that the C89 and C99 standards
suggest that `malloc(0)` really ought to return NULL.

|The Swiss-Army-knife aspect of realloc is daunting at first,
|but this interface rewards patient study. Soon you realize that
|zero-null realloc was thoughtfully designed to enable elegant
|dynamic arrays that do exactly the right thing under all
|circumstances, obviating the need for clunky and error-prone
|code to handle grow-from-zero and shrink-to-zero as special
|cases.

This seems specious, at best. Doug McIlroy implemented realloc
and this is the opposite of what he requested when it was
standardized:
https://www.tuhs.org/pipermail/tuhs/2015-September/007452.html

Note that the original author of `realloc` requested that
`realloc(0)` retun something other than NULL. That directly
contradicts the author's suggestion that `realloc` was
"thoughtfully designed to enable elegant dynamic arrays".

Moreover, the `realloc0` function that was discussed here does
not strike me as either "clunky" or "error-prone", and makes the
authors' code well-defined on any implementation. Given their
noxious code style, it's litearlly a one-line fix. Sure, I
suppose it's subjective?

|Figure 1 illustrates idiomatic realloc via a simple stack that
|grows with every push() and shrinks with every pop(). Pointer S
|and counter N (lines 1 and 2) represent the stack: S points to
|an array of N strictly positive ints. Because they are
|statically allocated, initially the pointer is NULL and the
|counter is zero, indicating an empty stack. Function resize
|(lines 4-10) resizes the stack to a given new capacity,
|checking for arithmetic overflow (line 6) before calling
|realloc and checking the return value for memory exhaustion
|(line 8). Allocation failure is inferred when a nonzero new
|size is requested but NULL is returned; zero-null realloc also
|returns NULL when the second argument is zero, but this does
|not indicate an allocation failure because no allocation was
|attempted. (Checking errno doesn't enable portable code to
|detect allocation failure because the C standards don't say
|how out-of-memory affects errno.) Thanks to zero-null realloc's
|versatility, the resize function need not consider whether the
|stack is growing from zero or shrinking to zero or re-sizing in
|some other way; everything Just Works regardless.

This paragraph is the root of my objection; particularly the
first sentence. This code is described as "idiomatic", but
according to whom? Moreover, that sentence doesn't explicitly
mention "zero-null realloc"; they defer discussion of that
until later in the paragraph (if it were me, I'd have prefaced
the entire thing with something like, "For implementations that
provide zero-null semantics, ..."). Critically, they _don't
describe what happens outside of their zero-null semantic
model_. Would replacing the `realloc` call in their code with
the ternary expression discussed earlier in this thread not be
"idiomatic"? Could an argument be made that it is even more
idiomatic to use an `if` statement here to guard against the
`realloc(0)` case?

They go on:

|The code of figure 1 follows a few simple rules implicit in the
|semantics of zero-null realloc. Functions push and pop (lines
|12-23) access the stack only via subscripts on S, because
|realloc may move the array to a different location in memory.
|They never dereference S when N is zero. The resize function
|resists the temptation of reckless S = realloc(S,...), which
|destroys the entry point into the array when allocation fails,
|thereby leaking memory and losing data.

It strikes me that every thing they mention in this paragraph is
equally applicable outside of the zero-null semantic model they
defined.

|I've been seeing code resembling figure 1 for 30 years,
|starting with the work of an older schoolmate who had bothered
|to read the fine manual; the clarity and simplicity of his code
|left a deep impression. In the decades since then I have
|repeatedly found idiomatic realloc in serious production code,
|usually while scanning for p = realloc(p,...) bugs.

What does this mean? What fine manual did the author's
schoolmate read? What does it mean to have used this code (or
something rather like it for 30 years) if not that it was
portable? Were the authors using the same platform the entire
time, with no upgrades? I very much doubt it.

|Imagine, then, my dismay when I learned that C23 declares
|realloc(ptr,0) to be undefined behavior, thereby pulling the
|rug out from under a widespread and exemplary pattern
|deliberately condoned by C89 through C11. So much for stare
|decisis. Compile idiomatic realloc code as C23 and the compiler
|might maul the source in most astonishing ways and your machine
|could ignite at runtime.

"...thereby pulling the rug out from under a widespread and
exemplary pattern deliberately condoned by C89 through C11."
Say what now? This pattern has been implementation defined-
defined, and thus always fraught, for decades now. The scenario
mentioned (upgrading a compiler or library) could blow the code
up with C11, too; granted, the failure case might not be so
dramatic as spontaneous combustion; recompilation or relinking
with an upgraded compiler or library could change to "zero
non-null" semantics and mask allocation failures.

I think a reasonable reader could see the language and use of
words like, "widespread", "exemplary" and "condoned" by earlier
standards, not to mention "idiomatic" and conclude the authors
suggest there is some level of portability to this code that is
not, in fact there. For the record, a reasonable reader could
conclude the opposite, as well; I merely suggest that it is
ambiguous.

When combined with their incorrect statements about the
motivation behind the design of `realloc` and the overall tone
of the article, I'm not giving them the benefit of the doubt.

I did talk to JeanHyde about the motivation, and it sure seemed
reasonable to me: the root issue has caused production failures
in the wild. In other words, code like the authors' "exemplary"
example caused real outages. Everyone agreed that the situation
with regard to making `realloc` UB sucked, but it was the best
of a set of bad options. The people on the committee are not
ignorant fools, after all.

- Dan C.

On Thu, 20 Jul 2023 22:35:33 -0000 (UTC), cross@spitfire.i.gajendra.net
(Dan Cross) wrote in <u9ccrl$i2q$1@reader2.panix.com>:

> (thankfully it has been
> edited since initially published to remove some of the more offensive
> language: the slur about "neurodivergent ideas" is gone)

I can attest to seeing that in one of the
earlier releases of the document.

I wonder if the ACM ever published a post
mortem / mea culpa for any of that...or did they sweep it
under the rug and hope nobody noticed?

I've done a smattering of C programming
since the 90's. I hope they don't ruin the language,
its stability is part of its appeal.

Learned C using "Learn C Now" from Microsoft in the 90's.
This was a C editor and compiler, but you couldn't
save the executables, only run them.

Anyway, many years later, while using gcc, I ran into a
problem: I was using string constants to allocate
string space, and gcc stopped allowing that by default.
"-fwriteable-strings I think is the flag to
allow it, but it got me thinking about safety
in software, and I learned to not be as
foolish with my programming.

So some changes to C are good,
sometimes -- just not sure these kids
can steer nowadays. ;)

--
-v

vallor <vallor@cultnix.org> writes:

> Anyway, many years later, while using gcc, I ran into a
> problem: I was using string constants to allocate
> string space, and gcc stopped allowing that by default.
> "-fwriteable-strings I think is the flag to
> allow it, but it got me thinking about safety
> in software, and I learned to not be as
> foolish with my programming.

Did you know you can write

char writable[] = "this string determines the size and initial value";

?

--
Ben.

vallor <vallor@cultnix.org> writes:
[...]
> Anyway, many years later, while using gcc, I ran into a
> problem: I was using string constants to allocate
> string space, and gcc stopped allowing that by default.
> "-fwriteable-strings I think is the flag to
> allow it, but it got me thinking about safety
> in software, and I learned to not be as
> foolish with my programming.

gcc removed the -fwritable-strings option in 2004. I don't know what
release that change appeared in.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On Sat, 22 Jul 2023 16:28:00 -0700, Keith Thompson
<Keith.S.Thompson+u@gmail.com> wrote in
<87edkzmsqn.fsf@nosuchdomain.example.com>:

> vallor <vallor@cultnix.org> writes:
> [...]
>> Anyway, many years later, while using gcc, I ran into a problem: I was
>> using string constants to allocate string space, and gcc stopped
>> allowing that by default.
>> "-fwriteable-strings I think is the flag to allow it, but it got me
>> thinking about safety in software, and I learned to not be as foolish
>> with my programming.
>
> gcc removed the -fwritable-strings option in 2004. I don't know what
> release that change appeared in.

guess I'm showing my age...started the company in 1994.

--
-v

On Sat, 22 Jul 2023 22:04:17 +0100, Ben Bacarisse <ben.usenet@bsb.me.uk>
wrote in <87wmyrmze6.fsf@bsb.me.uk>:

> vallor <vallor@cultnix.org> writes:
>
>> Anyway, many years later, while using gcc, I ran into a problem: I was
>> using string constants to allocate string space, and gcc stopped
>> allowing that by default.
>> "-fwriteable-strings I think is the flag to allow it, but it got me
>> thinking about safety in software, and I learned to not be as foolish
>> with my programming.
>
> Did you know you can write
>
> char writable[] = "this string determines the size and initial value";
>
> ?

<facepalm> Well, I do now...though I never
even thought to try it, as anything that seems like writing
to an actual string constant appears to
be a 3rd rail to me now...

I wrote a proof-of-concept program to compare behavior
for writing past bounds for strings declared in this way,
strdup()'ed, and malloc()'ed. valgrind segfaults on
it, but it compiles and runs on its own. (Would
be willing to post it with the caveat that its
behavior is undefined.)

--
-v

In article <u9gjin$3pvhi$1@dont-email.me>, vallor <vallor@cultnix.org> wrote:
>On Thu, 20 Jul 2023 22:35:33 -0000 (UTC), cross@spitfire.i.gajendra.net
>(Dan Cross) wrote in <u9ccrl$i2q$1@reader2.panix.com>:
>
>> (thankfully it has been
>> edited since initially published to remove some of the more offensive
>> language: the slur about "neurodivergent ideas" is gone)
>
>I can attest to seeing that in one of the
>earlier releases of the document.
>
>I wonder if the ACM ever published a post
>mortem / mea culpa for any of that...or did they sweep it
>under the rug and hope nobody noticed?

To my knowledge, no; it was just silently removed.

- Dan C.

cross@spitfire.i.gajendra.net (Dan Cross) wrote:

> In article <u9gjin$3pvhi$1@dont-email.me>, vallor <vallor@cultnix.org> wrote:
>> On Thu, 20 Jul 2023 22:35:33 -0000 (UTC), cross@spitfire.i.gajendra.net
>> (Dan Cross) wrote in <u9ccrl$i2q$1@reader2.panix.com>:
>>> [ https://queue.acm.org/detail.cfm?id=3588242 ]
>>> (thankfully it has been edited since initially published to remove
>>> some of the more offensive language: the slur about "neurodivergent
>>> ideas" is gone)
>>
>> I can attest to seeing that in one of the earlier releases of the document.
>>
>> I wonder if the ACM ever published a post mortem / mea culpa for any
>> of that...or did they sweep it under the rug and hope nobody noticed?
>
> To my knowledge, no; it was just silently removed.

Archive.org still has a snapshot from 2023-04-04 with that in it:

As C89 was taking shape, the neurodivergent notion of a
"zero-length object" was making the rounds: Proponents argued
that a non-null pointer to such an object should be returned for
zero-byte allocation requests.

https://web.archive.org/web/20230404195105/https://queue.acm.org/detail.cfm?id=3588242

But the 2023-04-09 version has been edited to remove it:

As C89 was taking shape, the notion of a "zero-length object"
was making the rounds: Proponents argued that a non-null pointer
to such an object should be returned for zero-byte allocation
requests.

https://web.archive.org/web/20230409001708/https://queue.acm.org/detail.cfm?id=3588242

On 2023-07-26, Otto J. Makela <om@iki.fi> wrote:
> cross@spitfire.i.gajendra.net (Dan Cross) wrote:
>
>> In article <u9gjin$3pvhi$1@dont-email.me>, vallor <vallor@cultnix.org> wrote:
>>> On Thu, 20 Jul 2023 22:35:33 -0000 (UTC), cross@spitfire.i.gajendra.net
>>> (Dan Cross) wrote in <u9ccrl$i2q$1@reader2.panix.com>:
>>>> [ https://queue.acm.org/detail.cfm?id=3588242 ]
>>>> (thankfully it has been edited since initially published to remove
>>>> some of the more offensive language: the slur about "neurodivergent
>>>> ideas" is gone)
>>>
>>> I can attest to seeing that in one of the earlier releases of the document.
>>>
>>> I wonder if the ACM ever published a post mortem / mea culpa for any
>>> of that...or did they sweep it under the rug and hope nobody noticed?
>>
>> To my knowledge, no; it was just silently removed.
>
> Archive.org still has a snapshot from 2023-04-04 with that in it:
>
> As C89 was taking shape, the neurodivergent notion of a
> "zero-length object" was making the rounds: Proponents argued
> that a non-null pointer to such an object should be returned for
> zero-byte allocation requests.
>
> https://web.archive.org/web/20230404195105/https://queue.acm.org/detail.cfm?id=3588242
>
> But the 2023-04-09 version has been edited to remove it:

Haha! Who is the neuro? He who can wrap his head around the concept of
zero, or he who struggles with zero length due to a learning disability?

Of course they removed it; ranting against a straightforward concept
that neurotypical grade school children can wrap their heads around made
them look like fools, separately from the remark also being offensive to
some people.

Note that they didn't replace "neurodivergent" with something else like
"puzzing", "odd", "counterintuitive" or any other mild pejorative;
it was just removed.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca