Martin Uecker <ma.uecker@gmail.com> writes:

[some unrelated passages removed]

> On Wednesday, August 16, 2023 at 6:06:43?AM UTC+2, Tim Rentsch wrote:
>
>> Martin Uecker <ma.u...@gmail.com> writes:

[...]

>>> One could still consider the idea that "indeterminate" is an
>>> abstract property that yields UB during read even for types
>>> that do not have trap representations. There is no wording
>>> in the C standard to support this, but I would not call this
>>> idea "fundamentally wrong". You are right that this is different
>>> to provenance provenance which is about values. What it would
>>> have in common with pointer provenance is that there is hidden
>>> state in the abstract machine associated with memory that
>>> is not part of the representation. With effective types there
>>> is another example of this.
>>
>> I understand that you want to consider a broader topic, and that,
>> in the realm of that broader topic, something like provenance
>> could have a role to play. I think it is worth responding to
>> that thesis, and am expecting to do so in a separate reply (or
>> new thread?) although probably not right away.
>
> I would love to hear your comments, because some people
> want to have such an abstract of "indeterminate" and
> some already believe that this is how the standard should
> be understood already today.

I've been thinking about this, and am close (I think) to having
something to say in response. Before I do that, thought, let me
ask this: what problem or problems are motivating the question?
What problems do you (or "some people") want to solve? I don't
want just examples here; I'm hoping to get a full list.

On 2023-08-17, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> Martin Uecker <ma.uecker@gmail.com> writes:
>
> [some unrelated passages removed]
>
>> On Wednesday, August 16, 2023 at 6:06:43?AM UTC+2, Tim Rentsch wrote:
>>
>>> Martin Uecker <ma.u...@gmail.com> writes:
>
> [...]
>
>>>> One could still consider the idea that "indeterminate" is an
>>>> abstract property that yields UB during read even for types
>>>> that do not have trap representations. There is no wording
>>>> in the C standard to support this, but I would not call this
>>>> idea "fundamentally wrong". You are right that this is different
>>>> to provenance provenance which is about values. What it would
>>>> have in common with pointer provenance is that there is hidden
>>>> state in the abstract machine associated with memory that
>>>> is not part of the representation. With effective types there
>>>> is another example of this.
>>>
>>> I understand that you want to consider a broader topic, and that,
>>> in the realm of that broader topic, something like provenance
>>> could have a role to play. I think it is worth responding to
>>> that thesis, and am expecting to do so in a separate reply (or
>>> new thread?) although probably not right away.
>>
>> I would love to hear your comments, because some people
>> want to have such an abstract of "indeterminate" and
>> some already believe that this is how the standard should
>> be understood already today.
>
> I've been thinking about this, and am close (I think) to having
> something to say in response. Before I do that, thought, let me
> ask this: what problem or problems are motivating the question?
> What problems do you (or "some people") want to solve? I don't
> want just examples here; I'm hoping to get a full list.

I'm all about the diagnosis. Even on machines in which all
representations are values, and therefore safe, a program whose external
effect or output depends on unintialized data, and is therefore
nondeterministic (a bad form of nondeterministic), is a repugnant
program.

I'd like to have clear rules which allow an implementation to
to go great depths to diagnose all such situations, while
remaining conforming. (The language agrees that those situations
are erroneous, granting the tools license to diagnose.)

At the same time, certain situations in which uninitialized data are
used in ways that don't have a visible effect, would be nuisance if they
generated diagnostics, the primary example being the copying of objects.
I would like it so that memcpy isn't magic. I want it so that the
programmer can write a bytewise memcpy which doesn't violate the
rules even if it moves uninitialized data.

I would like a model of uninitialized data which usefully lends itself
to different depths with different trade-offs, like complexity of
analysis and use of run-time resources. Limits should be imposed by
implementations (what cases they want to diagnose) rather than by the
model.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

On Thursday, August 17, 2023 at 9:08:48 AM UTC+2, Kaz Kylheku wrote:
> On 2023-08-17, Tim Rentsch <tr.1...@z991.linuxsc.com> wrote:
> > Martin Uecker <ma.u...@gmail.com> writes:
> >
> > [some unrelated passages removed]
> >
> >> On Wednesday, August 16, 2023 at 6:06:43?AM UTC+2, Tim Rentsch wrote:
> >>
> >>> Martin Uecker <ma.u...@gmail.com> writes:
> >
> > [...]
> >
> >>>> One could still consider the idea that "indeterminate" is an
> >>>> abstract property that yields UB during read even for types
> >>>> that do not have trap representations. There is no wording
> >>>> in the C standard to support this, but I would not call this
> >>>> idea "fundamentally wrong". You are right that this is different
> >>>> to provenance provenance which is about values. What it would
> >>>> have in common with pointer provenance is that there is hidden
> >>>> state in the abstract machine associated with memory that
> >>>> is not part of the representation. With effective types there
> >>>> is another example of this.
> >>>
> >>> I understand that you want to consider a broader topic, and that,
> >>> in the realm of that broader topic, something like provenance
> >>> could have a role to play. I think it is worth responding to
> >>> that thesis, and am expecting to do so in a separate reply (or
> >>> new thread?) although probably not right away.
> >>
> >> I would love to hear your comments, because some people
> >> want to have such an abstract of "indeterminate" and
> >> some already believe that this is how the standard should
> >> be understood already today.
> >
> > I've been thinking about this, and am close (I think) to having
> > something to say in response. Before I do that, thought, let me
> > ask this: what problem or problems are motivating the question?
> > What problems do you (or "some people") want to solve? I don't
> > want just examples here; I'm hoping to get a full list.
> I'm all about the diagnosis. Even on machines in which all
> representations are values, and therefore safe,

I do not agree with the idea that "absence of UB = safe ".

> a program whose external
> effect or output depends on unintialized data, and is therefore
> nondeterministic (a bad form of nondeterministic), is a repugnant
> program.

I would expect a debugger to output the memory as it seen
by the CPU. But yes, it would not be a strictly conforming program.

> I'd like to have clear rules which allow an implementation to
> to go great depths to diagnose all such situations, while
> remaining conforming. (The language agrees that those situations
> are erroneous, granting the tools license to diagnose.)

An implementation does not need a license from the standard
to diagnose anything. I can already diagnose whatever seems
useful and this does not affect conformance at all.

But it becomes easier to usefully diagnose behavior which is
undefined, because then one can expect that in portable C it
is not used intentionally.

> At the same time, certain situations in which uninitialized data are
> used in ways that don't have a visible effect, would be nuisance if they
> generated diagnostics, the primary example being the copying of objects.
> I would like it so that memcpy isn't magic. I want it so that the
> programmer can write a bytewise memcpy which doesn't violate the
> rules even if it moves uninitialized data.

Yes, I think for C this is rather important.

> I would like a model of uninitialized data which usefully lends itself
> to different depths with different trade-offs, like complexity of
> analysis and use of run-time resources. Limits should be imposed by
> implementations (what cases they want to diagnose) rather than by the
> model.

Tools can already do complex analysis and track down use of
uninitialized variables. But with respect to conformance, I think
the current standard has very good rules: memcpy/memcmp
and similar code works as expected. Locally, where a compiler
can be expected to give good diagnostics via static analysis
the use of uninitialized variables is UB. But this does not
spread via pointers elsewhere, where useful diagnostics
are unlikely and optimizer induced problems based on UB
might be far more difficult to debug.

Martin

Kaz Kylheku <864-117-4973@kylheku.com> writes:

> I'm all about the diagnosis. Even on machines in which all
> representations are values, and therefore safe, a program whose
> external effect or output depends on unintialized data, and is
> therefore nondeterministic (a bad form of nondeterministic), is a
> repugnant program.
>
> I'd like to have clear rules which allow an implementation to to
> go great depths to diagnose all such situations, while remaining
> conforming. (The language agrees that those situations are
> erroneous, granting the tools license to diagnose.)

The C standard allows compilers to do whatever analysis they
want and to issue diagnostics for whatever conditions or
circumstances they choose. What you want is orthogonal to
what is being discussed.

On 2023-08-19, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> Kaz Kylheku <864-117-4973@kylheku.com> writes:
>
>> I'm all about the diagnosis. Even on machines in which all
>> representations are values, and therefore safe, a program whose
>> external effect or output depends on unintialized data, and is
>> therefore nondeterministic (a bad form of nondeterministic), is a
>> repugnant program.
>>
>> I'd like to have clear rules which allow an implementation to to
>> go great depths to diagnose all such situations, while remaining
>> conforming. (The language agrees that those situations are
>> erroneous, granting the tools license to diagnose.)
>
> The C standard allows compilers to do whatever analysis they
> want and to issue diagnostics for whatever conditions or
> circumstances they choose.

And stop translating? If some use of an uninitialized object
isn't undefined, and you make the diagnostic a fatal error,
then you don't have a conforming compiler at that point.

> What you want is orthogonal to what is being discussed.

I'm mainly concerned about run-time.

If the program hasn't invoked undefined behavior, I don't thinkk it's
conforming to inject gratuitous diagnostics into the program's run-time,
such that they appear as if they were its output on stderr or stdout.
Those diagnostics have to go to some special debug port.

Also, not conforming to arbitrarily terminate the program. (Other
than in some weasly language lawyering way, by declaring that it
has exceeded an implementation limit or something.)

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Kaz Kylheku <864-117-4973@kylheku.com> writes:

> On 2023-08-19, Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:

[...]

>> The C standard allows compilers to do whatever analysis they
>> want and to issue diagnostics for whatever conditions or
>> circumstances they choose.
>
> And stop translating? If some use of an uninitialized object
> isn't undefined, and you make the diagnostic a fatal error,
> then you don't have a conforming compiler at that point.
>
> [also]
>
> If the program hasn't invoked undefined behavior, I don't thinkk
> it's conforming to inject gratuitous diagnostics [..or..]
> to arbitrarily terminate the program. [...]

You need to learn how to say what you mean. Your earlier
posting didn't say anything about failing to compile
or altering program behavior. If you can't learn how
to say what you mean then there is roughly a 1e-29 percent
chance that you'll get what you want.

On Saturday, August 19, 2023 at 7:04:10 AM UTC+2, Kaz Kylheku wrote:
> On 2023-08-18, Martin Uecker <ma.u...@gmail.com> wrote:
> > On Thursday, August 17, 2023 at 9:08:48 AM UTC+2, Kaz Kylheku wrote:
> > An implementation does not need a license from the standard
> > to diagnose anything. I can already diagnose whatever seems
> > useful and this does not affect conformance at all.
> That's true about diagnostics at translation time. It's not clear
> about that happen at run time and indistinguishable from the
> program's output on stdout or stderr.

The observable behavior has to stay the same, so yes, it could
not output to stdout or stderr. But there is nothing stopping it
to log debugging information somewhere else, where it could
be accessed.

> Also, it might be desirable for it to be conforming to terminate the
> program if it has run afoul of the rules.

Yes, this is one main reason to make certain things UB. But
then it can have false positives and needs to be backward
compatible, which limits what is possible.

> >> I would like a model of uninitialized data which usefully lends itself
> >> to different depths with different trade-offs, like complexity of
> >> analysis and use of run-time resources. Limits should be imposed by
> >> implementations (what cases they want to diagnose) rather than by the
> >> model.
> >
> > Tools can already do complex analysis and track down use of
> > uninitialized variables. But with respect to conformance, I think
> > the current standard has very good rules: memcpy/memcmp
> > and similar code works as expected. Locally, where a compiler
> > can be expected to give good diagnostics via static analysis
> > the use of uninitialized variables is UB. But this does not
> > spread via pointers elsewhere, where useful diagnostics
> > are unlikely and optimizer induced problems based on UB
> > might be far more difficult to debug.
> Dynamic instrumentation and tracking makes that possible
> for that information to follow pointer data flows, globally
> in the program.
>
> E.g. under the Valgrind tool, if one module passes an unitialized
> object into another, and that other one relies on it to make
> a conditional branch, it will be diagnosed. You can get the
> backtrace of where that object was created as well as where
> the use took place.

And valgrind exists and is a useful tool (I use it myself)
despite not everything it diagnoses is UB. But it also has
false positives, so using the same rules for deciding what
should be UB in the standard as valgrind uses seems difficult.

Also note that of the output of a program relies on
unspecified values, then it is already not strictly conforming
even when the behavior itself is not undefined. So if an
implementation is smart enough to see this, it could already
reject the program.

Making already the use of unspecified values in conditional
branches be UB seems problematic. E.g. you could not
compute a hash over data structures with padding and
then compare it later to see whether something has
changed (taking into account false positives). This seems
similar to memcpy / memcmp but involved conditions,
and such techniques would become non-conforming.

Martin

On 8/19/23 4:36 AM, Martin Uecker wrote:
> On Saturday, August 19, 2023 at 7:04:10 AM UTC+2, Kaz Kylheku wrote:
>> On 2023-08-18, Martin Uecker <ma.u...@gmail.com> wrote:
>>> On Thursday, August 17, 2023 at 9:08:48 AM UTC+2, Kaz Kylheku wrote:
>>> An implementation does not need a license from the standard
>>> to diagnose anything. I can already diagnose whatever seems
>>> useful and this does not affect conformance at all.
>> That's true about diagnostics at translation time. It's not clear
>> about that happen at run time and indistinguishable from the
>> program's output on stdout or stderr.
>
> The observable behavior has to stay the same, so yes, it could
> not output to stdout or stderr. But there is nothing stopping it
> to log debugging information somewhere else, where it could
> be accessed.
>
>> Also, it might be desirable for it to be conforming to terminate the
>> program if it has run afoul of the rules.
>
> Yes, this is one main reason to make certain things UB. But
> then it can have false positives and needs to be backward
> compatible, which limits what is possible.
>
>>>> I would like a model of uninitialized data which usefully lends itself
>>>> to different depths with different trade-offs, like complexity of
>>>> analysis and use of run-time resources. Limits should be imposed by
>>>> implementations (what cases they want to diagnose) rather than by the
>>>> model.
>>>
>>> Tools can already do complex analysis and track down use of
>>> uninitialized variables. But with respect to conformance, I think
>>> the current standard has very good rules: memcpy/memcmp
>>> and similar code works as expected. Locally, where a compiler
>>> can be expected to give good diagnostics via static analysis
>>> the use of uninitialized variables is UB. But this does not
>>> spread via pointers elsewhere, where useful diagnostics
>>> are unlikely and optimizer induced problems based on UB
>>> might be far more difficult to debug.
>> Dynamic instrumentation and tracking makes that possible
>> for that information to follow pointer data flows, globally
>> in the program.
>>
>> E.g. under the Valgrind tool, if one module passes an unitialized
>> object into another, and that other one relies on it to make
>> a conditional branch, it will be diagnosed. You can get the
>> backtrace of where that object was created as well as where
>> the use took place.
>
> And valgrind exists and is a useful tool (I use it myself)
> despite not everything it diagnoses is UB. But it also has
> false positives, so using the same rules for deciding what
> should be UB in the standard as valgrind uses seems difficult.
>
> Also note that of the output of a program relies on
> unspecified values, then it is already not strictly conforming
> even when the behavior itself is not undefined. So if an
> implementation is smart enough to see this, it could already
> reject the program.
>
> Making already the use of unspecified values in conditional
> branches be UB seems problematic. E.g. you could not
> compute a hash over data structures with padding and
> then compare it later to see whether something has
> changed (taking into account false positives). This seems
> similar to memcpy / memcmp but involved conditions,
> and such techniques would become non-conforming.
>
> Martin

My understanding is that there is no requirement that the values of the
padding bytes remains constant over time. I can't imagine a case where
they will just change at an arbitrary time, but setting a member of the
structure to a value (even if it is the same value it had) might easily
affect the value of the padding bytes, so the hash changes.

On Saturday, August 19, 2023 at 3:18:22 PM UTC+2, Richard Damon wrote:
> On 8/19/23 4:36 AM, Martin Uecker wrote:
> > On Saturday, August 19, 2023 at 7:04:10 AM UTC+2, Kaz Kylheku wrote:
> >> On 2023-08-18, Martin Uecker <ma.u...@gmail.com> wrote:
> >>> On Thursday, August 17, 2023 at 9:08:48 AM UTC+2, Kaz Kylheku wrote:
> >>> An implementation does not need a license from the standard
> >>> to diagnose anything. I can already diagnose whatever seems
> >>> useful and this does not affect conformance at all.
> >> That's true about diagnostics at translation time. It's not clear
> >> about that happen at run time and indistinguishable from the
> >> program's output on stdout or stderr.
> >
> > The observable behavior has to stay the same, so yes, it could
> > not output to stdout or stderr. But there is nothing stopping it
> > to log debugging information somewhere else, where it could
> > be accessed.
> >
> >> Also, it might be desirable for it to be conforming to terminate the
> >> program if it has run afoul of the rules.
> >
> > Yes, this is one main reason to make certain things UB. But
> > then it can have false positives and needs to be backward
> > compatible, which limits what is possible.
> >
> >>>> I would like a model of uninitialized data which usefully lends itself
> >>>> to different depths with different trade-offs, like complexity of
> >>>> analysis and use of run-time resources. Limits should be imposed by
> >>>> implementations (what cases they want to diagnose) rather than by the
> >>>> model.
> >>>
> >>> Tools can already do complex analysis and track down use of
> >>> uninitialized variables. But with respect to conformance, I think
> >>> the current standard has very good rules: memcpy/memcmp
> >>> and similar code works as expected. Locally, where a compiler
> >>> can be expected to give good diagnostics via static analysis
> >>> the use of uninitialized variables is UB. But this does not
> >>> spread via pointers elsewhere, where useful diagnostics
> >>> are unlikely and optimizer induced problems based on UB
> >>> might be far more difficult to debug.
> >> Dynamic instrumentation and tracking makes that possible
> >> for that information to follow pointer data flows, globally
> >> in the program.
> >>
> >> E.g. under the Valgrind tool, if one module passes an unitialized
> >> object into another, and that other one relies on it to make
> >> a conditional branch, it will be diagnosed. You can get the
> >> backtrace of where that object was created as well as where
> >> the use took place.
> >
> > And valgrind exists and is a useful tool (I use it myself)
> > despite not everything it diagnoses is UB. But it also has
> > false positives, so using the same rules for deciding what
> > should be UB in the standard as valgrind uses seems difficult.
> >
> > Also note that of the output of a program relies on
> > unspecified values, then it is already not strictly conforming
> > even when the behavior itself is not undefined. So if an
> > implementation is smart enough to see this, it could already
> > reject the program.
> >
> > Making already the use of unspecified values in conditional
> > branches be UB seems problematic. E.g. you could not
> > compute a hash over data structures with padding and
> > then compare it later to see whether something has
> > changed (taking into account false positives). This seems
> > similar to memcpy / memcmp but involved conditions,
> > and such techniques would become non-conforming.
> >
> > Martin
> My understanding is that there is no requirement that the values of the
> padding bytes remains constant over time.

The C standard specifies when they can change:

"When a value is stored in an object of structure or union type,
including in a member object, the bytes of the object representation
that correspond to any padding bytes take unspecified values"

> I can't imagine a case where
> they will just change at an arbitrary time, but setting a member of the
> structure to a value (even if it is the same value it had) might easily
> affect the value of the padding bytes, so the hash changes.

Sure, writing to object may change the padding and then the
hash changes. This is why I mentioned false positives.

Martin

Martin Uecker <ma.uecker@gmail.com> writes:

> On Thursday, August 17, 2023 at 8:13:07?AM UTC+2, Tim Rentsch wrote:
>
>> Martin Uecker <ma.u...@gmail.com> writes:
>>
>> [some unrelated passages removed]
>>
>>> On Wednesday, August 16, 2023 at 6:06:43?AM UTC+2, Tim Rentsch wrote:
>>>
>>>> Martin Uecker <ma.u...@gmail.com> writes:
>>
>> [...]
>>
>>>>> One could still consider the idea that "indeterminate" is an
>>>>> abstract property that yields UB during read even for types
>>>>> that do not have trap representations. There is no wording
>>>>> in the C standard to support this, but I would not call this
>>>>> idea "fundamentally wrong". You are right that this is different
>>>>> to provenance provenance which is about values. What it would
>>>>> have in common with pointer provenance is that there is hidden
>>>>> state in the abstract machine associated with memory that
>>>>> is not part of the representation. With effective types there
>>>>> is another example of this.
>>>>
>>>> I understand that you want to consider a broader topic, and that,
>>>> in the realm of that broader topic, something like provenance
>>>> could have a role to play. I think it is worth responding to
>>>> that thesis, and am expecting to do so in a separate reply (or
>>>> new thread?) although probably not right away.
>>>
>>> I would love to hear your comments, because some people
>>> want to have such an abstract of "indeterminate" and
>>> some already believe that this is how the standard should
>>> be understood already today.
>>
>> I've been thinking about this, and am close (I think) to having
>> something to say in response. Before I do that, thought, let me
>> ask this: what problem or problems are motivating the question?
>> What problems do you (or "some people") want to solve? I don't
>> want just examples here; I'm hoping to get a full list.
>
> There are essentially two main interests driving this. First,
> there is some interest to precisely formulate the semantics for C.
> The provenance proposal came out of this.
>
> Second, there is the issue of safety problems caused by
> uninitialized reads, together with compiler support for zero
> initialization etc. So there are various people who want to
> change the semantics for uninitialized variables completely
> in the interest of safety.

This response doesn't answer my question. What are the problems,
specifically, that people want to solve? If there isn't a good
understanding of what the problem is, there is little hope of
finding a solution, let alone reaching agreement on whether a
proposed change does in fact solve the problem. If we don't know
where we're going, any choice of road is equally good.

That said, I understand that you are asking not on your own behalf
but on behalf (perhaps indirectly) of others, and the others might
not know what the problem(s) are that they want to solve. I think
it's worth asking the question explicitly, What is the problem
that we want to solve here? Start by simply trying to write a
clear statement of what the problem is; proceed on to looking for
a solution only after there is agreement (and I don't mean just a
majority vote) about what problem it is the group wants to solve.

(Note added after writing: I didn't realize when I started how
difficult this subject is and how much there is to say about it.
I hope readers will appreciate the amount of effort that has
been invested, and get some value out of what has been produced,
even if it spends too much time on some less important issues.)

(Also, after having written the whole posting, I see that there
are some aspects that I didn't relate to the indeterminate
question and so didn't address. If you want me to say more about
formalizing semantics or the issue of safety for uninitialized
variables, I really need some specifics before I can talk about
those.)

(One further thought: on reading through my comments one last
time, I may have more to say about uninitialized variables. But
I am deferring that for now, to get this beast out the door.)

> So far, there was no consensus in WG14 that the rules should
> be changed or what the new rules should be.

That's because they don't know what problem it is that they want
to solve.

Consider the question of what happens with padding bits/bytes,
and unnamed members, in structs (unions too of course, but for
now we consider only structs). The C standard says these bits of
memory take unspecified values whenever there is a store to any
member of the struct (and maybe also at other times, but let's
ignore that). I understand why this decision was made, namely,
to give more freedom to implementations as to how such operations
are actualized. But it leaves behind a problem. Speaking as a
developer, I want the values of these bits to be stable, at least
in certain cases (and I want to be able to choose which cases
those are). The C language doesn't give me any way to do that,
at least not one that isn't horribly inconvenient. In making the
decision about padding bits/bytes, the C committee answered the
/question/ but didn't address the /problem/. I expect that
something similar is going on with the current discussions.

To better understand the landscape, let's look at three different
kinds of undefined behavior. The illustrating constructions are
signed integer arithmetic, obsolete pointer values, and violating
effective type rules.

Situations where arithmetic on signed integers overflows might be
called /practical/ undefined behavior. Certainly it would be
possible to require a better-defined semantics (such as giving an
unspecified result), but presumably overflow doesn't come up very
often, it's not clear how useful the "better" result would be,
and the cost in some hardware environments might be prohibitive.
Furthermore there is a fairly easy workaround to avoid overflow:
simply convert to unsigned types, do the operations, and then
convert back. Overflow being undefined behavior isn't absolutely
necessary but in practical terms it's acceptable. (I acknowledge
that some people have different views on that last statement.)

An obsolete pointer value is a pointer to an object after the end
of the object's lifetime. Attempting to make use of an obsolete
pointer value, in any way whatsoever including simply loading it
by means of lvalue conversion, is undefined behavior. We can
imagine narrowing the scope a bit so simply loading an obsolete
pointer value or comparing one for equality could be better
defined, but any attempt to dereference an obsolete pointer value
is what might be called an /essential/ undefined behavior. The
problem here is both practical and theoretical: there is no way
to be sure the underlying hardware will be able to carry out the
asked-for operation (without a machine check, etc), and even if
there were, there is no way to describe what happens in a way
that can be expressed (usefully) in terms that relate to what's
going on in the abstract machine. There simply is no practical,
useful, sensible way to define the behavior of dereferencing an
obsolete pointer value.

At the other end of the spectrum, violating effective type rules is
what might be called /gratuitous/ undefined behavior. There is no
particular hardware motivation for choosing UB. And there is no
problem defining the semantics of a cross-type access, which can be
done definedly in the same way as accessing union members. So there
is no reason to think that adding cross-type restrictions is
necessary. An argument can be made that cross-type restrictions
are /desirable/, because they allow code transformations that
improve performance in some cases.

Incidentally, it might seem like effective type rules are similar in
some way to NaT bits or pointer provenance. They aren't. NaT bits
are hardware indicators that actually exist, and pointer provenances
are attached to values, not to objects. Neither of those conditions
hold for effective types. The seeming similarity to hidden memory
bits is a red herring.

(Also, effective type rules are a lot more complicated than they
seem at first blush, and have some peculiar properties as a result.
They seem to work okay if not looked at too closely, but a closer
look shows some serious shortcomings. But I digress.)

There are two significant problems with undefined behavior. The
smaller of the two is that there are no distinctions between the
different classes of undefined behavior. There is no way around
having some sort of undefined behavior for obsolete pointer values,
but cross-typing rules are a completely different story. Yet the C
standard puts all the different kinds of undefined behaviors into
the same absolute category. Sometimes people use compiler options
to turn off, for example, so-called "strict aliasing", and of course
the C standard allows us to do that. But compilers aren't required
to provide such an option, and if they do the option may not do
exactly what we expect it to do, because there is no standard
specification for it. The C standard should define officially
sanctioned mechanisms -- as for example standard #pragma's -- to
give standard-defined semantics to certain constructs of undefined
behavior that resemble, eg, -fno-strict-aliasing.

On Sat, 26 Aug 2023 19:25:55 -0700
Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> Sometimes people use compiler options
> to turn off, for example, so-called "strict aliasing", and of course
> the C standard allows us to do that. But compilers aren't required
> to provide such an option, and if they do the option may not do
> exactly what we expect it to do, because there is no standard
> specification for it. The C standard should define officially
> sanctioned mechanisms -- as for example standard #pragma's -- to
> give standard-defined semantics to certain constructs of undefined
> behavior that resemble, eg, -fno-strict-aliasing.

Surely the starting point for this should be the documentation of the
compilers to specify precisely what -fno-strict-aliasing does. If
a consensus emerges out of these precise specifications or C programmers
indicate that they prefer the specification of some particular compiler
then this can become part of the standard. Adding a relevant #pragma
should be trivial.

> The second problem is basically The Law of Unintended Consequences
> smashing into The Law of Least Astonishment. As compiler writers
> have gotten more and more clever at exploiting the implications of
> "undefined behavior", we see more and more cases of code that looks
> reasonable being turned into mush by overly clever "optimizing"
> compilers. There is obviously something wrong with the way this
> trend is going -- ever more clever "optimizations", followed by ever
> more arcane compiler options to work around the problems caused by
> the too-clever compilers. This problem must be addressed by the C
> standard, for if it is not the ecosystem will transform into a
> confused state that is exactly what the C standard was put in place
> to avoid. (I do have some ideas about how to address this issue,
> but I want to make sure everyone appreciates the extent of the
> problem before we start talking about solutions.)

Without specific examples , it's impossible to comment on this. Why did
the "reasonable" code have the undefined behaviour ? Could the result
the programmer was aiming for have been achieved with defined behaviour
? For example it has been pointed out on comp.lang.c that it's
impossible to write a malloc() implementation in conforming C. This is
certainly a weakness which should be addressed with some appropriate
#pragma .

> Before leaving the sub-topic of undefined behavior, let me mention
> two success stories. The first is 'restrict': the performance
> implications are local, the choice is under control of the program
> (and programmer), and the default choice is to play safe. Good
> show.

From my point of view , restrict is not a success because the
specification of restrict is the one part of the C1999 standard I have
given up trying to understand. I understand the underlying idea but the
specifics elude me. I remember many years ago someone asked on this
group about some code involving restrict and a member of the standard
committee replied and I found the reply counterintuitive. So I have
decided to not use restrict in my own code taking also into account
that I don't need the microoptimisations which restrict is intended to
allow. But for all I know , people who do need these optimisations find
the specification of restrict in the standard perfectly adequate.

--
It is not widely known that the "CPC" in "Amstrad CPC" actually stands
for "cool people club".

Spiros Bousbouras <spibou@gmail.com> writes:

> On Sat, 26 Aug 2023 19:25:55 -0700
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> Sometimes people use compiler options to turn off, for example,
>> so-called "strict aliasing", and of course the C standard allows
>> us to do that. But compilers aren't required to provide such an
>> option, and if they do the option may not do exactly what we
>> expect it to do, because there is no standard specification for
>> it. The C standard should define officially sanctioned
>> mechanisms -- as for example standard #pragma's -- to give
>> standard-defined semantics to certain constructs of undefined
>> behavior that resemble, eg, -fno-strict-aliasing.
>
> Surely the starting point for this should be the documentation of
> the compilers to specify precisely what -fno-strict-aliasing does.
> [...]

Not at all. It's easy to write a specification that says what we
want to do, along similar lines to what is said in the footnote
about union member access in section 6.5.2.3

If the member used to access the contents of a union object
is not the same as the member last used to store a value in
the object, the appropriate part of the object representation
of the value is reinterpreted as an object representation in
the new type as described in 6.2.6 (a process sometimes called
"type punning"). This might be a trap representation.

That behavior should be the default, for all accesses. For cases
where a developer wants to give permission to the compiler to
optimize based on cross-type non-interference assumptions, there
should be a #pragma to do something similar to what effective type
rules do now. The effective type rules are in need of re-writing
anyway, and making type punning be the default doesn't break any
programs, because compilers are already free to ignore the
implications of violating effective type conditions.

>> The second problem is basically The Law of Unintended Consequences
>> smashing into The Law of Least Astonishment. As compiler writers
>> have gotten more and more clever at exploiting the implications of
>> "undefined behavior", we see more and more cases of code that looks
>> reasonable being turned into mush by overly clever "optimizing"
>> compilers. There is obviously something wrong with the way this
>> trend is going -- ever more clever "optimizations", followed by
>> ever more arcane compiler options to work around the problems
>> caused by the too-clever compilers. This problem must be addressed
>> by the C standard, for if it is not the ecosystem will transform
>> into a confused state that is exactly what the C standard was put
>> in place to avoid. (I do have some ideas about how to address this
>> issue, but I want to make sure everyone appreciates the extent of
>> the problem before we start talking about solutions.)
>
> Without specific examples , it's impossible to comment on this.
> [...]

I feel that so much has been written about this issue that it
isn't necessary for me to elaborate.

> For example it has been pointed out on comp.lang.c that it's
> impossible to write a malloc() implementation in conforming
> C. This is certainly a weakness which should be addressed with
> some appropriate #pragma .

There isn't any reason to think malloc() should be writable in
completely portable C. That's the point of putting malloc() in
the system library in the first place. By the way, with type
punning semantics mentioned above being the default, and with the
alignment features added in C11, I think it is possible to write
malloc() in portable C without needed any additional language
changes. But even if it isn't that is no cause for concern; one
of the principal reasons for having a system library is to
provide functionality that the core language cannot express (or
cannot express conveniently).

>> Before leaving the sub-topic of undefined behavior, let me mention
>> two success stories. The first is 'restrict': the performance
>> implications are local, the choice is under control of the program
>> (and programmer), and the default choice is to play safe. Good
>> show.
>
> From my point of view , restrict is not a success because the
> specification of restrict is the one part of the C1999 standard I
> have given up trying to understand. I understand the underlying
> idea but the specifics elude me. [...]

I agree the formal definition of restrict is rather daunting. In
practice though I think using restrict with confidence is not
overly difficult. My working model for restrict is something
like this:

1. Use restrict only in the declarations of function
parameters.

2. For a declaration like const T *restrict foo ,
the compiler may assume that any objects that can be
accessed through 'foo' will not be modified.

3. For a declaration like T *restrict bas ,
the compiler may assume that any changes to objects
that can be accessed through 'bas' will be done
using 'bas' or a pointer value derived from 'bas'
(and in particular that no changes will happen
other than through 'bas' or 'bas'-derived pointer
values).

Is this summary description helpful?

On Tue, 29 Aug 2023 04:35:40 -0700
Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> Spiros Bousbouras <spibou@gmail.com> writes:
>
> > On Sat, 26 Aug 2023 19:25:55 -0700
> > Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> >
> >> Sometimes people use compiler options to turn off, for example,
> >> so-called "strict aliasing", and of course the C standard allows
> >> us to do that. But compilers aren't required to provide such an
> >> option, and if they do the option may not do exactly what we
> >> expect it to do, because there is no standard specification for
> >> it. The C standard should define officially sanctioned
> >> mechanisms -- as for example standard #pragma's -- to give
> >> standard-defined semantics to certain constructs of undefined
> >> behavior that resemble, eg, -fno-strict-aliasing.
> >
> > Surely the starting point for this should be the documentation of
> > the compilers to specify precisely what -fno-strict-aliasing does.
> > [...]
>
> Not at all. It's easy to write a specification that says what we
> want to do, along similar lines to what is said in the footnote
> about union member access in section 6.5.2.3
>
> If the member used to access the contents of a union object
> is not the same as the member last used to store a value in
> the object, the appropriate part of the object representation
> of the value is reinterpreted as an object representation in
> the new type as described in 6.2.6 (a process sometimes called
> "type punning"). This might be a trap representation.

Works for me but it would be good to know that this is how compiler
writers actually understand -fno-strict-aliasing .Is there any compiler
documentation which says something like this ?

> That behavior should be the default, for all accesses. For cases
> where a developer wants to give permission to the compiler to
> optimize based on cross-type non-interference assumptions, there
> should be a #pragma to do something similar to what effective type
> rules do now. The effective type rules are in need of re-writing
> anyway, and making type punning be the default doesn't break any
> programs, because compilers are already free to ignore the
> implications of violating effective type conditions.

[...]

> > For example it has been pointed out on comp.lang.c that it's
> > impossible to write a malloc() implementation in conforming
> > C. This is certainly a weakness which should be addressed with
> > some appropriate #pragma .
>
> There isn't any reason to think malloc() should be writable in
> completely portable C. That's the point of putting malloc() in
> the system library in the first place. By the way, with type
> punning semantics mentioned above being the default, and with the
> alignment features added in C11, I think it is possible to write
> malloc() in portable C without needed any additional language
> changes. But even if it isn't that is no cause for concern; one
> of the principal reasons for having a system library is to
> provide functionality that the core language cannot express (or
> cannot express conveniently).

One might want to experiment with different allocation algorithms
and it seems to me that this sort of thing is within the "remit" of
C. So ideally one should be able to write it in C and prove , starting
from the standard or precise specifications in compiler documentation ,
that it works correctly. I don't necessarily mean prove the correctness
of the whole code but certain key parts.

Another application I have in mind is languages which get translated
to C and support garbage collection. Again one might want to use the
standard malloc() to allocate a large block of memory and use different
parts of this memory for different types of objects.

If with the semantics you propose these things are possible , I'm happy.
I'm not bothered which is the default as long as there is a precise
specification from which you can reason that you get the desired behaviour.

> >> Before leaving the sub-topic of undefined behavior, let me mention
> >> two success stories. The first is 'restrict': the performance
> >> implications are local, the choice is under control of the program
> >> (and programmer), and the default choice is to play safe. Good
> >> show.
> >
> > From my point of view , restrict is not a success because the
> > specification of restrict is the one part of the C1999 standard I
> > have given up trying to understand. I understand the underlying
> > idea but the specifics elude me. [...]
>
> I agree the formal definition of restrict is rather daunting. In
> practice though I think using restrict with confidence is not
> overly difficult. My working model for restrict is something
> like this:
>
> 1. Use restrict only in the declarations of function
> parameters.
>
> 2. For a declaration like const T *restrict foo ,
> the compiler may assume that any objects that can be
> accessed through 'foo' will not be modified.

Wouldn't that also be the case with just const T * foo ?

> 3. For a declaration like T *restrict bas ,
> the compiler may assume that any changes to objects
> that can be accessed through 'bas' will be done
> using 'bas' or a pointer value derived from 'bas'
> (and in particular that no changes will happen
> other than through 'bas' or 'bas'-derived pointer
> values).
>
> Is this summary description helpful?

It seems clear enough but , as I've said , I don't have any use for
restrict anyway and it's not worth it for me to expend the additional
mental effort to confirm that my code obeys the additional restrictions
of restrict .If I call a function with a preexisting interface which
involves restrict then it seems easy enough to obey the restrictions.

--
Carrie also narrates the film, providing useful guidelines for those
challenged by its intricacies. Sample: "Later that day, Big and I
arrived home."
http://www.rogerebert.com/reviews/sex-and-the-city-2-2010

Spiros Bousbouras <spibou@gmail.com> writes:

> On Tue, 29 Aug 2023 04:35:40 -0700
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>
>> Spiros Bousbouras <spibou@gmail.com> writes:
>>
>>> On Sat, 26 Aug 2023 19:25:55 -0700
>>> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
>>>
>>>> Sometimes people use compiler options to turn off, for example,
>>>> so-called "strict aliasing", and of course the C standard allows
>>>> us to do that. But compilers aren't required to provide such an
>>>> option, and if they do the option may not do exactly what we
>>>> expect it to do, because there is no standard specification for
>>>> it. The C standard should define officially sanctioned
>>>> mechanisms -- as for example standard #pragma's -- to give
>>>> standard-defined semantics to certain constructs of undefined
>>>> behavior that resemble, eg, -fno-strict-aliasing.
>>>
>>> Surely the starting point for this should be the documentation of
>>> the compilers to specify precisely what -fno-strict-aliasing does.
>>> [...]
>>
>> Not at all. It's easy to write a specification that says what we
>> want to do, along similar lines to what is said in the footnote
>> about union member access in section 6.5.2.3
>>
>> If the member used to access the contents of a union object
>> is not the same as the member last used to store a value in
>> the object, the appropriate part of the object representation
>> of the value is reinterpreted as an object representation in
>> the new type as described in 6.2.6 (a process sometimes called
>> "type punning"). This might be a trap representation.
>
> Works for me but it would be good to know that this is how compiler
> writers actually understand -fno-strict-aliasing . [...]

No, it wouldn't. Implementations follow the C standard, not
the other way around. Looking at what implementations do for
the -fno-strict-aliasing flag is worse than a waste of time.

>>> For example it has been pointed out on comp.lang.c that it's
>>> impossible to write a malloc() implementation in conforming
>>> C. This is certainly a weakness which should be addressed with
>>> some appropriate #pragma .
>>
>> There isn't any reason to think malloc() should be writable in
>> completely portable C. That's the point of putting malloc() in
>> the system library in the first place. By the way, with type
>> punning semantics mentioned above being the default, and with the
>> alignment features added in C11, I think it is possible to write
>> malloc() in portable C without needed any additional language
>> changes. But even if it isn't that is no cause for concern; one
>> of the principal reasons for having a system library is to
>> provide functionality that the core language cannot express (or
>> cannot express conveniently).
>
> One might want to experiment with different allocation algorithms
> and it seems to me that this sort of thing is within the "remit" of
> C. So ideally one should be able to write it in C [...]

You're conflating writing something in C and writing something
in completely portable C. It's already possible to do these
things writing in C.

>>> From my point of view , restrict is not a success because the
>>> specification of restrict is the one part of the C1999 standard I
>>> have given up trying to understand. I understand the underlying
>>> idea but the specifics elude me. [...]
>>
>> I agree the formal definition of restrict is rather daunting. In
>> practice though I think using restrict with confidence is not
>> overly difficult. My working model for restrict is something
>> like this:
>>
>> 1. Use restrict only in the declarations of function
>> parameters.
>>
>> 2. For a declaration like const T *restrict foo ,
>> the compiler may assume that any objects that can be
>> accessed through 'foo' will not be modified.
>
> Wouldn't that also be the case with just const T * foo ?

No.

>> 3. For a declaration like T *restrict bas ,
>> the compiler may assume that any changes to objects
>> that can be accessed through 'bas' will be done
>> using 'bas' or a pointer value derived from 'bas'
>> (and in particular that no changes will happen
>> other than through 'bas' or 'bas'-derived pointer
>> values).
>>
>> Is this summary description helpful?
>
> It seems clear enough but , as I've said , I don't have any use
> for restrict anyway and it's not worth it for me to expend the
> additional mental effort to confirm that my code obeys the
> additional restrictions of restrict. [...]

If you don't want to use restrict that is quite okay. Part of
why I call restrict a success is that it can be ignored, with
only minimal effort, by any developer who doesn't want to use it.

On Wed, 30 Aug 2023 17:40:52 -0700
Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> Spiros Bousbouras <spibou@gmail.com> writes:
>
> > On Tue, 29 Aug 2023 04:35:40 -0700
> > Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:
> >
> >> Spiros Bousbouras <spibou@gmail.com> writes:

[...]

> >> Not at all. It's easy to write a specification that says what we
> >> want to do, along similar lines to what is said in the footnote
> >> about union member access in section 6.5.2.3
> >>
> >> If the member used to access the contents of a union object
> >> is not the same as the member last used to store a value in
> >> the object, the appropriate part of the object representation
> >> of the value is reinterpreted as an object representation in
> >> the new type as described in 6.2.6 (a process sometimes called
> >> "type punning"). This might be a trap representation.
> >
> > Works for me but it would be good to know that this is how compiler
> > writers actually understand -fno-strict-aliasing . [...]
>
> No, it wouldn't. Implementations follow the C standard, not
> the other way around. Looking at what implementations do for
> the -fno-strict-aliasing flag is worse than a waste of time.

Actually the influence goes in both directions. In theory the standard is the
ultimate authority , in practice whatever C compilers one has access to. For
now the standard doesn't have something like -fno-strict-aliasing so if one
needs it then looking at what implementations do is the only option. But even
the standard committee should look at it and whether C programmers find it
useful to decide what around such lines (if anything) should go into the
standard.

> >> There isn't any reason to think malloc() should be writable in
> >> completely portable C. That's the point of putting malloc() in
> >> the system library in the first place. By the way, with type
> >> punning semantics mentioned above being the default, and with the
> >> alignment features added in C11, I think it is possible to write
> >> malloc() in portable C without needed any additional language
> >> changes. But even if it isn't that is no cause for concern; one
> >> of the principal reasons for having a system library is to
> >> provide functionality that the core language cannot express (or
> >> cannot express conveniently).
> >
> > One might want to experiment with different allocation algorithms
> > and it seems to me that this sort of thing is within the "remit" of
> > C. So ideally one should be able to write it in C [...]
>
> You're conflating writing something in C and writing something
> in completely portable C. It's already possible to do these
> things writing in C.

I wrote

One might want to experiment with different allocation algorithms and it
seems to me that this sort of thing is within the "remit" of C. So
ideally one should be able to write it in C and prove , starting from the
standard or precise specifications in compiler documentation , that it
works correctly. I don't necessarily mean prove the correctness of the
whole code but certain key parts.

..This doesn't conflate anything. One can do the writing but can one do the
proving or something close ?

--
vlaho.ninja/prog

Spiros Bousbouras <spibou@gmail.com> writes:

> On Wed, 30 Aug 2023 17:40:52 -0700
> Tim Rentsch <tr.17687@z991.linuxsc.com> wrote:

[...]

>> You're conflating writing something in C and writing something
>> in completely portable C. It's already possible to do these
>> things writing in C.
>
> I wrote
>
> One might want to experiment with different allocation
> algorithms and it seems to me that this sort of thing is
> within the "remit" of C. So ideally one should be able to
> write it in C and prove , starting from the standard or
> precise specifications in compiler documentation , that it
> works correctly. I don't necessarily mean prove the
> correctness of the whole code but certain key parts.
>
> .This doesn't conflate anything. One can do the writing but
> can one do the proving or something close ?

A substitute for malloc()/free() can be written in standard C.

A substitute for malloc()/free() can not be written in completely
portable standard C.

I hope this clarifies my earlier comments.

Martin Uecker <ma.uecker@gmail.com> writes:

[...]

> There are essentially two main interests driving this. First,
> there is some interest to precisely formulate the semantics for
> C. The provenance proposal came out of this.
>
> Second, there is the issue of safety problems caused by
> uninitialized reads, together with compiler support for zero
> initialization etc. So there are various people who want to
> change the semantics for uninitialized variables completely
> in the interest of safety.
>
> So far, there was no consensus in WG14 that the rules should
> be changed or what the new rules should be.

I have a second reply here, which I hope will come closer to
being relevant to the issues of interest.

What I think is being looked for is a way to describe the
language semantics in areas such as cross-type interference and
what is meant when an uninitialized object is read. I thought
about this question both while I was writing the longer earlier
reply and then more deeply afterwards.

What I think is most important is that these areas in particular
are not about language semantics in the same way as, for example,
array indexing. Rather they are about what transformations a
compiler is allowed to do in the presence of various combinations
of program constructs. That difference means the C standard
should express the rules in a way that more directly reflects
what's going on. More specifically, the standard should say or
explain what can be done, not by describing language semantics
(which is indirect), but explicitly in terms of what compiler
transformations are allowed (which is direct). Note that there
is precedent for this idea, in how the C standard talks about
looping constructs and when they may be assumed to terminate.

To give an example, take uninitialized objects, either automatic
variables without an initializer, or memory allocated by malloc or
added by realloc. The most natural semantics for such situations
is to say that newly "created" memory gets an unspecified object
representation at the start of its lifetime. (Yes I know that C
in its current form lets automatic objects be "uninitialized"
whenever their declaration points are reached, but let's ignore
that for now.) Now suppose a program has a read access where it
is easy to deduce that the object being read is still in the
"unspecified object representation" initial state. To simplify
the discussion, suppose the type of the access is a pointer type,
and so is known to have trap representations (the name is changed
in the C23 draft, but the idea is what's important).

What is a compiler allowed to do in such circumstances? One thing
it might reasonably be allowed to do is to cause the program to be
terminated if it ever reaches such an access. Or there might be
an option to initialize the pointer to NULL. Or, if a suitable
compiler option were invoked, the construct might be flagged with
a fatal error (or of course a warning). There are all sorts of
actions a developer might want the compiler to take, and a
compiler could offer many of those options, as choices selected
under control of command line switches (or equivalent). I think a
few points are worth making.

One, there must be some sort of default action that all compilers
have to support. The default action in this case might be to
issue a non-fatal diagnostic.

Two, there must be a way for the developer to tell the compiler to
"proceed blindly" - saying, in effect, I accept that the compiled
code might misbehave, but let me take that risk, and generate code
like it's going to work. (In other words, for the read access, go
ahead and load whatever unspecified object representation happens
to be there.) A "proceed blindly" choice probably shouldn't be
the default, but it must be available.

Three, the consequence must never be "undefined behavior", unless
there is an explicit stipulation to that effect. The stipulation
might take the form of a #pragma, or a compiler option, or a code
decoration using "attribute" (whatever the syntax for such things
is).

I know my comments here are somewhat sketchy, but hopefully a
general sense of the ideas gets across. The suggestions should at
least serve to stimulate further discussion.

On 2023-09-06 02:03, Tim Rentsch wrote:
> Martin Uecker <ma.uecker@gmail.com> writes:
>
> [...]
>
>> There are essentially two main interests driving this. First,
>> there is some interest to precisely formulate the semantics for
>> C. The provenance proposal came out of this.
>>
>> Second, there is the issue of safety problems caused by
>> uninitialized reads, together with compiler support for zero
>> initialization etc. So there are various people who want to
>> change the semantics for uninitialized variables completely
>> in the interest of safety.
>>
>> So far, there was no consensus in WG14 that the rules should
>> be changed or what the new rules should be.
>
> I have a second reply here, which I hope will come closer to
> being relevant to the issues of interest.
>
> What I think is being looked for is a way to describe the
> language semantics in areas such as cross-type interference and
> what is meant when an uninitialized object is read. I thought
> about this question both while I was writing the longer earlier
> reply and then more deeply afterwards.
>
> What I think is most important is that these areas in particular
> are not about language semantics in the same way as, for example,
> array indexing. Rather they are about what transformations a
> compiler is allowed to do in the presence of various combinations
> of program constructs. That difference means the C standard
> should express the rules in a way that more directly reflects
> what's going on. More specifically, the standard should say or
> explain what can be done, not by describing language semantics
> (which is indirect), but explicitly in terms of what compiler
> transformations are allowed (which is direct). Note that there
> is precedent for this idea, in how the C standard talks about
> looping constructs and when they may be assumed to terminate.
>
> To give an example, take uninitialized objects, either automatic
> variables without an initializer, or memory allocated by malloc or
> added by realloc. The most natural semantics for such situations
> is to say that newly "created" memory gets an unspecified object
> representation at the start of its lifetime. (Yes I know that C
> in its current form lets automatic objects be "uninitialized"
> whenever their declaration points are reached, but let's ignore
> that for now.) Now suppose a program has a read access where it
> is easy to deduce that the object being read is still in the
> "unspecified object representation" initial state. To simplify
> the discussion, suppose the type of the access is a pointer type,
> and so is known to have trap representations (the name is changed
> in the C23 draft, but the idea is what's important).
>
> What is a compiler allowed to do in such circumstances? One thing
> it might reasonably be allowed to do is to cause the program to be
> terminated if it ever reaches such an access. Or there might be
> an option to initialize the pointer to NULL. Or, if a suitable
> compiler option were invoked, the construct might be flagged with
> a fatal error (or of course a warning). There are all sorts of
> actions a developer might want the compiler to take, and a
> compiler could offer many of those options, as choices selected
> under control of command line switches (or equivalent). I think a
> few points are worth making.
>
> One, there must be some sort of default action that all compilers
> have to support. The default action in this case might be to
> issue a non-fatal diagnostic.
>
> Two, there must be a way for the developer to tell the compiler to
> "proceed blindly" - saying, in effect, I accept that the compiled
> code might misbehave, but let me take that risk, and generate code
> like it's going to work. (In other words, for the read access, go
> ahead and load whatever unspecified object representation happens
> to be there.) A "proceed blindly" choice probably shouldn't be
> the default, but it must be available.
>
> Three, the consequence must never be "undefined behavior", unless
> there is an explicit stipulation to that effect. The stipulation
> might take the form of a #pragma, or a compiler option, or a code
> decoration using "attribute" (whatever the syntax for such things
> is).
>

Agreed so far!

As a developer of programs in C with practical but not infinite
portability, I very much abhore the mad optimizations that use
language lawyering to state that any code path that might,
hypothetically, exceed the boundaries of standard-enforced behavior
is allowed to be arbitrarily mangled to get a faster bad result.

For example, I have one function which intentionally reads an
uninitialized variable to get a somewhat arbitrary value of a type
with no known trap representation. I have a number of other
programs which extensively process a block of data before deciding
in some other way if the data is garbage or useful. This is done
for sound technical reasons but requires that the compiler doesn't
plant landmines all over virgin land.

As another example, I have speed critical code that relies on running
on 2s complement machines with wraparound on signed integer overflow,
and that code is being very clear and explicit in doing so, but there
is no C90 notation to tell all ISO-C implementation that this is the
intention, thus it is explicit only in comments, not in the tokens
passed to the C compiler.

> I know my comments here are somewhat sketchy, but hopefully a
> general sense of the ideas gets across. The suggestions should at
> least serve to stimulate further discussion.
>

I am writing from a similar perspective .

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Jakob Bohm <jb-usenet@wisemo.com.invalid> writes:

> As another example, I have speed critical code that relies on running
> on 2s complement machines with wraparound on signed integer overflow, and
> that code is being very clear and explicit in doing so, but there
> is no C90 notation to tell all ISO-C implementation that this is the
> intention, thus it is explicit only in comments, not in the tokens
> passed to the C compiler.

You can tell the compiler you want 2s complement by using the intN_t
types if you can find one that suits your portability requirements.

And can you not use unsigned arithmetic, re-interpreting as signed for
those places where it matters? The "overflow" can only happen in
the arithmetic, not in the re-interpretation.

I know this is a deviation from the topic, so feel free to ignore if you
don't want to get into it.

--
Ben.

On 2023-09-07 18:19, Ben Bacarisse wrote:
> Jakob Bohm <jb-usenet@wisemo.com.invalid> writes:
>
>> As another example, I have speed critical code that relies on running
>> on 2s complement machines with wraparound on signed integer overflow, and
>> that code is being very clear and explicit in doing so, but there
>> is no C90 notation to tell all ISO-C implementation that this is the
>> intention, thus it is explicit only in comments, not in the tokens
>> passed to the C compiler.
>
> You can tell the compiler you want 2s complement by using the intN_t
> types if you can find one that suits your portability requirements.
>
> And can you not use unsigned arithmetic, re-interpreting as signed for
> those places where it matters? The "overflow" can only happen in
> the arithmetic, not in the re-interpretation.
>
> I know this is a deviation from the topic, so feel free to ignore if you
> don't want to get into it.
>

The code in question has as explicit design condition that the compiler
implements signed versions with wraparound for each unsigned int type .

The code cannot rely on the intN_t types because they were not part of
C90 and thus do not exist as separate types in some targeted compilers.

In the world of C90 compilers, stdint.h was a non-standard system header
that provided convenience names for the most closely matching C90 types
on the platform, and some platforms simply didn't provide that header,
instead documenting how each C90 type mapped to data sizes.

Excessive casting where directly using the desired type seems possible
is highly counter-intuitive and thus it is inherently wrong for an
optimizer to presume the right to mangle code using types such as "int",
"short int", "long int" and "signed char".

Once again this comes down to a language drift from "undefined" meaning
"not defined by this standard" to "An extremely toxic trap condition" .

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Jakob Bohm <jb-usenet@wisemo.com.invalid> writes:

> On 2023-09-07 18:19, Ben Bacarisse wrote:
>> Jakob Bohm <jb-usenet@wisemo.com.invalid> writes:
>>
>>> As another example, I have speed critical code that relies on running
>>> on 2s complement machines with wraparound on signed integer overflow, and
>>> that code is being very clear and explicit in doing so, but there
>>> is no C90 notation to tell all ISO-C implementation that this is the
>>> intention, thus it is explicit only in comments, not in the tokens
>>> passed to the C compiler.
>> You can tell the compiler you want 2s complement by using the intN_t
>> types if you can find one that suits your portability requirements.
>> And can you not use unsigned arithmetic, re-interpreting as signed for
>> those places where it matters? The "overflow" can only happen in
>> the arithmetic, not in the re-interpretation.
>> I know this is a deviation from the topic, so feel free to ignore if you
>> don't want to get into it.
>
> The code in question has as explicit design condition that the compiler
> implements signed versions with wraparound for each unsigned int type .
>
> The code cannot rely on the intN_t types because they were not part of
> C90 and thus do not exist as separate types in some targeted
> compilers.

Ah, I didn't know targetting C90 was still a thing. I've been out of
the business for many years.

> Excessive casting where directly using the desired type seems possible
> is highly counter-intuitive and thus it is inherently wrong for an
> optimizer to presume the right to mangle code using types such as "int",
> "short int", "long int" and "signed char".

I wasn't suggesting casts as they don't remove the undefined behaviour.
But you have a design that suits your needs so it's all good.

--
Ben.

On Thursday, August 17, 2023 at 8:13:07 AM UTC+2, Tim Rentsch wrote:
> Martin Uecker <ma.u...@gmail.com> writes:
>
> [some unrelated passages removed]
> > On Wednesday, August 16, 2023 at 6:06:43?AM UTC+2, Tim Rentsch wrote:
> >
> >> Martin Uecker <ma.u...@gmail.com> writes:
> [...]
> >>> One could still consider the idea that "indeterminate" is an
> >>> abstract property that yields UB during read even for types
> >>> that do not have trap representations. There is no wording
> >>> in the C standard to support this, but I would not call this
> >>> idea "fundamentally wrong". You are right that this is different
> >>> to provenance provenance which is about values. What it would
> >>> have in common with pointer provenance is that there is hidden
> >>> state in the abstract machine associated with memory that
> >>> is not part of the representation. With effective types there
> >>> is another example of this.
> >>
> >> I understand that you want to consider a broader topic, and that,
> >> in the realm of that broader topic, something like provenance
> >> could have a role to play. I think it is worth responding to
> >> that thesis, and am expecting to do so in a separate reply (or
> >> new thread?) although probably not right away.
> >
> > I would love to hear your comments, because some people
> > want to have such an abstract of "indeterminate" and
> > some already believe that this is how the standard should
> > be understood already today.
> I've been thinking about this, and am close (I think) to having
> something to say in response. Before I do that, thought, let me
> ask this: what problem or problems are motivating the question?
> What problems do you (or "some people") want to solve? I don't
> want just examples here; I'm hoping to get a full list.

There are essentially two main interests driving this. First, there
is some interest to precisely formulate the semantics for C.
The provenance proposal came out of this.

Second, there is the issue of safety problems caused by
uninitialized reads, together with compiler support for zero
initialization etc. So there are various people who want to
change the semantics for uninitialized variables completely
in the interest of safety.

So far, there was no consensus in WG14 that the rules should
be changed or what the new rules should be.

Martin

On 2023-08-18, Martin Uecker <ma.uecker@gmail.com> wrote:
> On Thursday, August 17, 2023 at 9:08:48 AM UTC+2, Kaz Kylheku wrote:
> An implementation does not need a license from the standard
> to diagnose anything. I can already diagnose whatever seems
> useful and this does not affect conformance at all.

That's true about diagnostics at translation time. It's not clear
about that happen at run time and indistinguishable from the
program's output on stdout or stderr.

Also, it might be desirable for it to be conforming to terminate the
program if it has run afoul of the rules.

>> I would like a model of uninitialized data which usefully lends itself
>> to different depths with different trade-offs, like complexity of
>> analysis and use of run-time resources. Limits should be imposed by
>> implementations (what cases they want to diagnose) rather than by the
>> model.
>
> Tools can already do complex analysis and track down use of
> uninitialized variables. But with respect to conformance, I think
> the current standard has very good rules: memcpy/memcmp
> and similar code works as expected. Locally, where a compiler
> can be expected to give good diagnostics via static analysis
> the use of uninitialized variables is UB. But this does not
> spread via pointers elsewhere, where useful diagnostics
> are unlikely and optimizer induced problems based on UB
> might be far more difficult to debug.

Dynamic instrumentation and tracking makes that possible
for that information to follow pointer data flows, globally
in the program.

E.g. under the Valgrind tool, if one module passes an unitialized
object into another, and that other one relies on it to make
a conditional branch, it will be diagnosed. You can get the
backtrace of where that object was created as well as where
the use took place.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca