On 7/30/23 7:10 AM, Bart wrote:
> On 30/07/2023 04:52, Richard Damon wrote:
> > On 7/29/23 9:00 PM, Keith Thompson wrote:
>
> >> I haven't actually used UTF-8 string literals.  As of C11/N1570, the
> >> standard is a bit vague about the difference.
> >>
> >> N1570 6.4.5p3:
> >>
> >>      A *character string literal* is a sequence of zero or more
> multibyte
> >>      characters enclosed in double-quotes, as in "xyz". A *UTF−8 string
> >>      literal* is the same, except prefixed by u8.
> >>
> >> p6:
> >>
> >>      For character string literals, the array elements have type char,
> >>      and are initialized with the individual bytes of the multibyte
> >>      character sequence. For UTF−8 string literals, the array elements
> >>      have type char, and are initialized with the characters of the
> >>      multibyte character sequence, as encoded in UTF−8.
> >>
> >> I would guess that something like u8"\xff", which specifies an invalid
> >> UTF-8 sequence, would be invalid, but it doesn't seem to violate any
> >> constraint or syntax rule, and neither gcc nor clang complains about
> it.
> >>
> >> I guess that for a compiler that uses EBCDIC for source code, "x" would
> >> be equivalent to "\xa7" (the EBCDIC code for 'x') and u8"x" would be
> >> equivalent to "\x78".
> >>
> >> Either the standard is insufficiently clear, or I'm missing something.
> >> I definitely wouldn't bet against the latter.
> >>
> >
> > I think the key is that numerical constant specified characters are
> > specified in the "execution character encoding", which is implementation
> > defined, and doesn't need to be unicode.
> >
> > Thus u8"\0xFF" might generate the byte sequence 0xC3 0xDF 0x00 to
> > represent the Unicode character U+00FF
>
> (Do you mean \0xFF or \xFF ?)
>
> I thought these were limited to 2 hex digits, but apparently you can
> have as many as you like. However that looks to be ambiguous:
>
>     "\x20AC"
>
> Is that a 4-digit hex code, or is it a 2-digit one following by normal
> characters A and C?

Yes, forgot that the 0 wasn't needed there.

As for the ambiguity, the grammar is defined by "maximum munching" so
"\x20AC" has a single "character" which takes at least 14 bits to
represent (so might not be a valid character). If you want the latter,
you need to write:

"\x20""AC" so that the constant is terminated and then in a later
phase the strings are concatenated. (a trap for machine generated code
that isn't smart enough to handle the case.)

On 30/07/2023 15:27, Richard Damon wrote:
> On 7/30/23 7:10 AM, Bart wrote:
>> On 30/07/2023 04:52, Richard Damon wrote:
>> > On 7/29/23 9:00 PM, Keith Thompson wrote:
>>
>> >> I haven't actually used UTF-8 string literals. As of C11/N1570, the
>> >> standard is a bit vague about the difference.
>> >>
>> >> N1570 6.4.5p3:
>> >>
>> >> A *character string literal* is a sequence of zero or more
>> multibyte
>> >> characters enclosed in double-quotes, as in "xyz". A *UTF−8
>> string
>> >> literal* is the same, except prefixed by u8.
>> >>
>> >> p6:
>> >>
>> >> For character string literals, the array elements have type
>> char,
>> >> and are initialized with the individual bytes of the multibyte
>> >> character sequence. For UTF−8 string literals, the array
>> elements
>> >> have type char, and are initialized with the characters of the
>> >> multibyte character sequence, as encoded in UTF−8.
>> >>
>> >> I would guess that something like u8"\xff", which specifies an
>> invalid
>> >> UTF-8 sequence, would be invalid, but it doesn't seem to violate any
>> >> constraint or syntax rule, and neither gcc nor clang complains
>> about it.
>> >>
>> >> I guess that for a compiler that uses EBCDIC for source code, "x"
>> would
>> >> be equivalent to "\xa7" (the EBCDIC code for 'x') and u8"x" would be
>> >> equivalent to "\x78".
>> >>
>> >> Either the standard is insufficiently clear, or I'm missing
>> something.
>> >> I definitely wouldn't bet against the latter.
>> >>
>> >
>> > I think the key is that numerical constant specified characters are
>> > specified in the "execution character encoding", which is
>> implementation
>> > defined, and doesn't need to be unicode.
>> >
>> > Thus u8"\0xFF" might generate the byte sequence 0xC3 0xDF 0x00 to
>> > represent the Unicode character U+00FF
>>
>> (Do you mean \0xFF or \xFF ?)
>>
>> I thought these were limited to 2 hex digits, but apparently you can
>> have as many as you like. However that looks to be ambiguous:
>>
>> "\x20AC"
>>
>> Is that a 4-digit hex code, or is it a 2-digit one following by normal
>> characters A and C?
>
> Yes, forgot that the 0 wasn't needed there.
>
> As for the ambiguity, the grammar is defined by "maximum munching" so
> "\x20AC" has a single "character" which takes at least 14 bits to
> represent (so might not be a valid character). If you want the latter,
> you need to write:

20AC should be Unicode for €. I assumed your example took Unicode
character codes, which need up to 21 bits, and turned them into UTF8
sequences.

However when I tried \x20AC with gcc 10.x, it said the hex escape
sequence was out of range, whether I used 'u8' or not.

> "\x20""AC" so that the constant is terminated and then in a later
> phase the strings are concatenated. (a trap for machine generated code
> that isn't smart enough to handle the case.)

Then it looks like "" can always be used to terminate a `\dd` sequence,
unless another \ follows anyway.

On Sunday, 30 July 2023 at 15:27:32 UTC+1, Richard Damon wrote:
> On 7/30/23 7:10 AM, Bart wrote:
> > On 30/07/2023 04:52, Richard Damon wrote:
> > > On 7/29/23 9:00 PM, Keith Thompson wrote:
> >
> > >> I haven't actually used UTF-8 string literals. As of C11/N1570, the
> > >> standard is a bit vague about the difference.
> > >>
> > >> N1570 6.4.5p3:
> > >>
> > >> A *character string literal* is a sequence of zero or more
> > multibyte
> > >> characters enclosed in double-quotes, as in "xyz". A *UTF−8 string
> > >> literal* is the same, except prefixed by u8.
> > >>
> > >> p6:
> > >>
> > >> For character string literals, the array elements have type char,
> > >> and are initialized with the individual bytes of the multibyte
> > >> character sequence. For UTF−8 string literals, the array elements
> > >> have type char, and are initialized with the characters of the
> > >> multibyte character sequence, as encoded in UTF−8.
> > >>
> > >> I would guess that something like u8"\xff", which specifies an invalid
> > >> UTF-8 sequence, would be invalid, but it doesn't seem to violate any
> > >> constraint or syntax rule, and neither gcc nor clang complains about
> > it.
> > >>
> > >> I guess that for a compiler that uses EBCDIC for source code, "x" would
> > >> be equivalent to "\xa7" (the EBCDIC code for 'x') and u8"x" would be
> > >> equivalent to "\x78".
> > >>
> > >> Either the standard is insufficiently clear, or I'm missing something.
> > >> I definitely wouldn't bet against the latter.
> > >>
> > >
> > > I think the key is that numerical constant specified characters are
> > > specified in the "execution character encoding", which is implementation
> > > defined, and doesn't need to be unicode.
> > >
> > > Thus u8"\0xFF" might generate the byte sequence 0xC3 0xDF 0x00 to
> > > represent the Unicode character U+00FF
> >
> > (Do you mean \0xFF or \xFF ?)
> >
> > I thought these were limited to 2 hex digits, but apparently you can
> > have as many as you like. However that looks to be ambiguous:
> >
> > "\x20AC"
> >
> > Is that a 4-digit hex code, or is it a 2-digit one following by normal
> > characters A and C?
> Yes, forgot that the 0 wasn't needed there.
>
> As for the ambiguity, the grammar is defined by "maximum munching" so
> "\x20AC" has a single "character" which takes at least 14 bits to
> represent (so might not be a valid character). If you want the latter,
> you need to write:
>
> "\x20""AC" so that the constant is terminated and then in a later
> phase the strings are concatenated. (a trap for machine generated code
> that isn't smart enough to handle the case.)
>
That a weakness in the Baby X resource compiler. It allows the user to enter
a C-style escaped string like "hello world\n". But it won't handle hex escapes.

On 7/30/23 11:05 AM, Bart wrote:
> On 30/07/2023 15:27, Richard Damon wrote:
> > On 7/30/23 7:10 AM, Bart wrote:
> >> On 30/07/2023 04:52, Richard Damon wrote:
> >>  > On 7/29/23 9:00 PM, Keith Thompson wrote:
> >>
> >>  >> I haven't actually used UTF-8 string literals.  As of C11/N1570,
> the
> >>  >> standard is a bit vague about the difference.
> >>  >>
> >>  >> N1570 6.4.5p3:
> >>  >>
> >>  >>      A *character string literal* is a sequence of zero or more
> >> multibyte
> >>  >>      characters enclosed in double-quotes, as in "xyz". A *UTF−8
> >> string
> >>  >>      literal* is the same, except prefixed by u8.
> >>  >>
> >>  >> p6:
> >>  >>
> >>  >>      For character string literals, the array elements have type
> >> char,
> >>  >>      and are initialized with the individual bytes of the multibyte
> >>  >>      character sequence. For UTF−8 string literals, the array
> >> elements
> >>  >>      have type char, and are initialized with the characters of the
> >>  >>      multibyte character sequence, as encoded in UTF−8.
> >>  >>
> >>  >> I would guess that something like u8"\xff", which specifies an
> >> invalid
> >>  >> UTF-8 sequence, would be invalid, but it doesn't seem to violate
> any
> >>  >> constraint or syntax rule, and neither gcc nor clang complains
> >> about it.
> >>  >>
> >>  >> I guess that for a compiler that uses EBCDIC for source code, "x"
> >> would
> >>  >> be equivalent to "\xa7" (the EBCDIC code for 'x') and u8"x"
> would be
> >>  >> equivalent to "\x78".
> >>  >>
> >>  >> Either the standard is insufficiently clear, or I'm missing
> >> something.
> >>  >> I definitely wouldn't bet against the latter.
> >>  >>
> >>  >
> >>  > I think the key is that numerical constant specified characters are
> >>  > specified in the "execution character encoding", which is
> >> implementation
> >>  > defined, and doesn't need to be unicode.
> >>  >
> >>  > Thus u8"\0xFF" might generate the byte sequence 0xC3 0xDF 0x00 to
> >>  > represent the Unicode character U+00FF
> >>
> >> (Do you mean \0xFF or \xFF ?)
> >>
> >> I thought these were limited to 2 hex digits, but apparently you can
> >> have as many as you like. However that looks to be ambiguous:
> >>
> >>      "\x20AC"
> >>
> >> Is that a 4-digit hex code, or is it a 2-digit one following by normal
> >> characters A and C?
> >
> > Yes, forgot that the 0 wasn't needed there.
> >
> > As for the ambiguity, the grammar is defined by "maximum munching" so
> > "\x20AC" has a single "character" which takes at least 14 bits to
> > represent (so might not be a valid character). If you want the latter,
> > you need to write:
>
> 20AC should be Unicode for €. I assumed your example took Unicode
> character codes, which need up to 21 bits, and turned them into UTF8
> sequences.
>

Someone earlier (not me) was asking about €, note that example made the
assumption that € was a character defined in both the source and
execution character sets (which don't need to be Unicode)

> However when I tried \x20AC with gcc 10.x, it said the hex escape
> sequence was out of range, whether I used 'u8' or not.

20AC is € only if the "Execution Character Set" is defined to be UNICODE.

I don't think GCC defines its "Execution Character Set" as UNICODE, at
least not for "narrow" characters, so that value isn't valid.

If you want to specify a Unicode character, you would use "\u20AC".
Note, the the execution character set can't reprent that character, you
might want u8"\u20AC" to get a UTF-8 string that represents that character.

This goes back to the fact that C vastly predates Unicode being a common
format, and C wants to allow implementations to use what ever is
actually natural for that system. It ALLOWS, but doesn't REQUIRE plain
strings to be a representation of it. Later version have made
understanding Unicode (as a possible additional representation) pretty
much a requirement for full conformance, but don't make plain char need
to be part of it.
>
>
> >   "\x20""AC" so that the constant is terminated and then in a later
> > phase the strings are concatenated. (a trap for machine generated code
> > that isn't smart enough to handle the case.)
>
> Then it looks like "" can always be used to terminate a `\dd` sequence,
> unless another \ follows anyway.

yes, because the first one ends the string that contained the \x escape
sequence, and then the second one starts another, and consecutive string
literals are automatically concatinated in a later phase.

On 7/29/23 23:52, Richard Damon wrote:
> Thus u8"\0xFF" might generate the byte sequence 0xC3 0xDF 0x00 to
> represent the Unicode character U+00FF

The "\0" is an octal escape sequence representing the null character.
That makes the 'x' and the two 'F's into ordinary characters, not part
of an escape sequence. I'm sure that's not what you intended.

Bart <bc@freeuk.com> writes:
> On 30/07/2023 04:52, Richard Damon wrote:
[...]
>> Thus u8"\0xFF" might generate the byte sequence 0xC3 0xDF 0x00 to
>> represent the Unicode character U+00FF
>
> (Do you mean \0xFF or \xFF ?)

\xFF is correct.

> I thought these were limited to 2 hex digits, but apparently you can
> have as many as you like. However that looks to be ambiguous:
>
> "\x20AC"
>
> Is that a 4-digit hex code, or is it a 2-digit one following by normal
> characters A and C?

N1570 6.4.4.4 (or the corresponding section on character constants in
whatever standard or draft you have).

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

Richard Damon <Richard@Damon-Family.org> writes:
[...]
> I don't think GCC defines its "Execution Character Set" as UNICODE, at
> least not for "narrow" characters, so that value isn't valid.

It does. From the gcc manual (version 11.3.0):

'-fexec-charset=CHARSET'
Set the execution character set, used for string and character
constants. The default is UTF-8. CHARSET can be any encoding
supported by the system's 'iconv' library routine.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

> Richard Damon <Richard@Damon-Family.org> writes:
> [...]
>
>> I don't think GCC defines its "Execution Character Set" as UNICODE, at
>> least not for "narrow" characters, so that value isn't valid.
>
> It does. From the gcc manual (version 11.3.0):
>
> '-fexec-charset=CHARSET'
> Set the execution character set, used for string and character
> constants. The default is UTF-8. CHARSET can be any encoding
> supported by the system's 'iconv' library routine.

The name of the option is something of a misnomer. Unicode is a
character set. UTF-8 is a particular encoding of Unicode.[*] The
description even says as much: "CHARSET can be any encoding ...".
A C implementation could choose Unicode for its execution character
set, but choose a different encoding to represent it.

In any case, thank you, it is nice to know about the option.

[*] I suppose UTF-8 could be viewed as an encoding of any set of
integer values taken from the domain of Unicode code points, but
undoubtedly the most common is to represent Unicode characters.

Richard Damon <Richard@Damon-Family.org> writes:

> On 7/30/23 11:05 AM, Bart wrote:

[concering a UTF-8 string literal u8"\x20AC"]

>> However when I tried \x20AC with gcc 10.x, it said the hex escape
>> sequence was out of range, whether I used 'u8' or not.
>
> [u8"\x20AC" is (something)] only if the "Execution Character Set"
> is defined to be UNICODE.

I believe this case is simply an oversight in the C standard.
Starting near the end of 2019, the C2x draft clearly establishes
that octal and hexadecimal escape sequences for u8 string literals
(and also u8 character constants) must have a value in the range of
unsigned char. The string literal u8"\x20AC" is meant to be a
constraint violation (and is one in C2x drafts)[*]. That result
doesn't depend on what the execution character set is.

[*] Assuming CHAR_BIT == 8, of course.

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

[.. utf-8 string literals ..]

> I haven't actually used UTF-8 string literals. As of C11/N1570,
> the standard is a bit vague about the difference.
>
> N1570 6.4.5p3:
>
> A *character string literal* is a sequence of zero or more
> multibyte characters enclosed in double-quotes, as in "xyz".
> A *UTF-8 string literal* is the same, except prefixed by u8.
>
> p6:
>
> For character string literals, the array elements have type
> char, and are initialized with the individual bytes of the
> multibyte character sequence. For UTF-8 string literals, the
> array elements have type char, and are initialized with the
> characters of the multibyte character sequence, as encoded in
> UTF?8.
>
> I would guess that something like u8"\xff", which specifies an invalid
> UTF-8 sequence, would be invalid, but it doesn't seem to violate any
> constraint or syntax rule, and neither gcc nor clang complains about it.

In C11, the source string is converted to the execution character
set, which is then represented using a UTF-8 encoding. 5.1.1.2 p5
in N1570 says:

Each source character set member and escape sequence in
character constants and string literals is converted to the
corresponding member of the execution character set; if
there is no corresponding member, it is converted to an
implementation-defined member other than the null (wide)
character.

Note: "Each source character set member and escape sequence" is
converted to the corresponding member of the execution character
set. I don't see anything that changes that rule for UTF-8
string literals.

I think this behavior is changing in C23, but I don't know
what the new wording means.

Richard Damon <Richard@Damon-Family.org> writes:

> As I understand it, a u8"string" will convert the string from its
> source character set (which might be set to Latin-1) into UTF-8,
> while a plain "string" will convert it to what is defined as the
> "execution" character set, which might be defined as the same as
> the input set of Latin-1.

For both u8-prefixed string literals and unprefixed string
literals, the source characters are first converted to the
corresponding members of the execution character set (or to
non-null implementation-defined members of the ECS in cases
where there is no corresponding member).

For u8-prefixed string literals, the sequence of execution
character set characters is then turned into a sequence of
bytes using a UTF-8 representation.

The execution character set is just a set. How characters
are represented as a sequence of multibyte characters is
independent of what characters are in the set.

Notice that source character set characters are converted into
their corresponding execution set members in translation phase 5,
per 5.1.1.2 paragraph 5, and the array value for the string
literal with a UTF-8 encoding is produced in translation phase 7,
per 6.4.5 paragraph 6 (references from N1570). Conversion to the
execution character set always comes first, for both u8-prefixed
string literals and unprefixed string literals.