BGB <cr88192@gmail.com> writes:
>On 5/9/2023 11:22 AM, Anton Ertl wrote:
>> Just do it, and you will see that it's much less painful than
>> converting between UTF-8 and UTF-32 all the time.
>>
>
>There is no straightforward mapping between UTF-8 bytes and the logical
>X position of a cursor on screen.
>
>One would likely need a loop or similar to walk the text to figure out
>where to put the cursor at.

True. But that is already true for plain ASCII, thanks to characters
like TAB. When you then go to UTF-32, you get things like zero-width
spaces, double-wide (on fixed-width fonts) characters (typically CJK)
and combining marks. So UTF-32 buys you absolutely nothing compared
to UTF-8.

You suggest having a 64-bit cell that represents a single glyph.
That's possible, but it's not UTF-32. You can get there from UTF-8
roughly as easily as from UTF-32. It solves the problem with
combining marks, but you still have glyphs with widths 0, 1, or 2, and
the TAB whose width depends on the stuff that came before it.

>If Emacs can handle huge files, maybe that is in its merit, but this is
>not likely a common use-case.

I have not made a survey of Emacs users, but I would not be surprised
if many users used it on huge files. I doubt I am the only one who
uses emacs to search, e.g., his mbox file. Another usage of huge
files has been to generate a trace of a program execution and then
search through it with Emacs.

Of course, if your tools don't support a particular usage, you won't
use them that way, and you will think that nobody does.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>BGB <cr88192@gmail.com> writes:
>>On 5/9/2023 11:22 AM, Anton Ertl wrote:
>>> Just do it, and you will see that it's much less painful than
>>> converting between UTF-8 and UTF-32 all the time.

>>If Emacs can handle huge files, maybe that is in its merit, but this is
>>not likely a common use-case.
>
>I have not made a survey of Emacs users, but I would not be surprised
>if many users used it on huge files. I doubt I am the only one who
>uses emacs to search, e.g., his mbox file. Another usage of huge
>files has been to generate a trace of a program execution and then
>search through it with Emacs.

Even Emacs has trouble with large program traces. As does Vim
(although opening it read-only helps a bit).

Really, split and grep are more useful for large traces ime.

On 5/11/2023 9:33 AM, Anton Ertl wrote:
> BGB <cr88192@gmail.com> writes:
>> On 5/9/2023 11:22 AM, Anton Ertl wrote:
>>> Just do it, and you will see that it's much less painful than
>>> converting between UTF-8 and UTF-32 all the time.
>>>
>>
>> There is no straightforward mapping between UTF-8 bytes and the logical
>> X position of a cursor on screen.
>>
>> One would likely need a loop or similar to walk the text to figure out
>> where to put the cursor at.
>
> True. But that is already true for plain ASCII, thanks to characters
> like TAB. When you then go to UTF-32, you get things like zero-width
> spaces, double-wide (on fixed-width fonts) characters (typically CJK)
> and combining marks. So UTF-32 buys you absolutely nothing compared
> to UTF-8.
>

The tab would usually become 4 or 8 space-like characters, which are
understood to represent a tab character.

> You suggest having a 64-bit cell that represents a single glyph.
> That's possible, but it's not UTF-32. You can get there from UTF-8
> roughly as easily as from UTF-32. It solves the problem with
> combining marks, but you still have glyphs with widths 0, 1, or 2, and
> the TAB whose width depends on the stuff that came before it.
>

I wasn't really arguing for UTF-32 in the first place.

>> If Emacs can handle huge files, maybe that is in its merit, but this is
>> not likely a common use-case.
>
> I have not made a survey of Emacs users, but I would not be surprised
> if many users used it on huge files. I doubt I am the only one who
> uses emacs to search, e.g., his mbox file. Another usage of huge
> files has been to generate a trace of a program execution and then
> search through it with Emacs.
>
> Of course, if your tools don't support a particular usage, you won't
> use them that way, and you will think that nobody does.
>

OK.

> - anton

BGB <cr88192@gmail.com> writes:
>On 5/11/2023 9:33 AM, Anton Ertl wrote:
>> True. But that is already true for plain ASCII, thanks to characters
>> like TAB. When you then go to UTF-32, you get things like zero-width
>> spaces, double-wide (on fixed-width fonts) characters (typically CJK)
>> and combining marks. So UTF-32 buys you absolutely nothing compared
>> to UTF-8.
>>
>
>The tab would usually become 4 or 8 space-like characters, which are
>understood to represent a tab character.

Tabs don't work that way. With tab stops at columns divisible by 8, a
tab can be equivalent to 1-8 spaces; and how many depends on the stuff
earlier in the line.

>I wasn't really arguing for UTF-32 in the first place.

One less reason to have anything but UTF-8.

- anton
--
'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
Mitch Alsup, <c17fcd89-f024-40e7-a594-88a85ac10d20o@googlegroups.com>

On Thursday, May 11, 2023 at 12:36:50 PM UTC-5, Anton Ertl wrote:
> BGB <cr8...@gmail.com> writes:
> >On 5/11/2023 9:33 AM, Anton Ertl wrote:
> >> True. But that is already true for plain ASCII, thanks to characters
> >> like TAB. When you then go to UTF-32, you get things like zero-width
> >> spaces, double-wide (on fixed-width fonts) characters (typically CJK)
> >> and combining marks. So UTF-32 buys you absolutely nothing compared
> >> to UTF-8.
> >>
> >
> >The tab would usually become 4 or 8 space-like characters, which are
> >understood to represent a tab character.
<
> Tabs don't work that way. With tab stops at columns divisible by 8, a
> tab can be equivalent to 1-8 spaces; and how many depends on the stuff
> earlier in the line.
<
And then there is how tabs work on a O29 key punch: You place a card in
the hopper and punch marks on the columns you want tabs to stop. You
then remove the card, and wrap it around the <forgot term> and stick this
back into the key punch.
<
Henceforth, every time you key a tab, the card slides over to the next stop..
<
Essentially equivalent to "tabs 5 12 15 27 38 49 64" for example.
<
FORTRAN formatting tabs are entirely different in that one can tab-backwards...
<
> >I wasn't really arguing for UTF-32 in the first place.
> One less reason to have anything but UTF-8.
> - anton
> --
> 'Anyone trying for "industrial quality" ISA should avoid undefined behavior.'
> Mitch Alsup, <c17fcd89-f024-40e7...@googlegroups.com>

On Thursday, May 11, 2023 at 9:59:21 AM UTC-5, Anton Ertl wrote:
>
> True. But that is already true for plain ASCII, thanks to characters
> like TAB. When you then go to UTF-32, you get things like zero-width
> spaces, double-wide (on fixed-width fonts) characters (typically CJK)
> and combining marks. So UTF-32 buys you absolutely nothing compared
> to UTF-8.
<
Why would you want zero-width spaces ?

On 5/11/2023 12:29 PM, Anton Ertl wrote:
> BGB <cr88192@gmail.com> writes:
>> On 5/11/2023 9:33 AM, Anton Ertl wrote:
>>> True. But that is already true for plain ASCII, thanks to characters
>>> like TAB. When you then go to UTF-32, you get things like zero-width
>>> spaces, double-wide (on fixed-width fonts) characters (typically CJK)
>>> and combining marks. So UTF-32 buys you absolutely nothing compared
>>> to UTF-8.
>>>
>>
>> The tab would usually become 4 or 8 space-like characters, which are
>> understood to represent a tab character.
>
> Tabs don't work that way. With tab stops at columns divisible by 8, a
> tab can be equivalent to 1-8 spaces; and how many depends on the stuff
> earlier in the line.
>

Granted, was assuming the case of tabs at the start of the line.
But, a variable number isn't that much different than 4 or 8, one just
sort of needs some special characters to designate them as representing
a tab.

>> I wasn't really arguing for UTF-32 in the first place.
>
> One less reason to have anything but UTF-8.
>

For external storage or sending them from one place to another, granted.

Putting UTF-32 in files or similar would be pointless and wasteful.
Similar for using them in OS API calls, ...

There is an edge case for systems which fall in the "Java legacy", which
tend to assume UTF-16 for a lot of stuff internally.

This leading to the whole "pretend string is UTF16 but store it
internally as 1252 if it maps directly" thing.

But, yeah, if designing a new language, might make more sense to assume
UTF-8 or similar for strings.

> - anton

MitchAlsup <MitchAlsup@aol.com> writes:
> On Thursday, May 11, 2023 at 9:59:21 AM UTC-5, Anton Ertl wrote:
>> True. But that is already true for plain ASCII, thanks to characters
>> like TAB. When you then go to UTF-32, you get things like zero-width
>> spaces, double-wide (on fixed-width fonts) characters (typically CJK)
>> and combining marks. So UTF-32 buys you absolutely nothing compared
>> to UTF-8.
> <
> Why would you want zero-width spaces ?

https://en.wikipedia.org/wiki/Zero-width_space

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */

According to MitchAlsup <MitchAlsup@aol.com>:
>On Thursday, May 11, 2023 at 9:59:21 AM UTC-5, Anton Ertl wrote:
>>
>> True. But that is already true for plain ASCII, thanks to characters
>> like TAB. When you then go to UTF-32, you get things like zero-width
>> spaces, double-wide (on fixed-width fonts) characters (typically CJK)
>> and combining marks. So UTF-32 buys you absolutely nothing compared
>> to UTF-8.
><
>Why would you want zero-width spaces ?

They show the boundaries between words in scripts that don't use
explicit spaces, and they're useful in Latin text to show places where
you can break a long word. There are also Zero Width Joiner and Zero
Width Nonjoiner codes which are useful in languages like Arabic where
adjacent characters are sometimes combined except when they aren't.

Pretty much anything you think you know about text layout is wrong
in at least some of the languages people use in computer text.

--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

On 11/05/2023 20:18, John Levine wrote:
> According to MitchAlsup <MitchAlsup@aol.com>:
>> On Thursday, May 11, 2023 at 9:59:21 AM UTC-5, Anton Ertl wrote:
>>>
>>> True. But that is already true for plain ASCII, thanks to characters
>>> like TAB. When you then go to UTF-32, you get things like zero-width
>>> spaces, double-wide (on fixed-width fonts) characters (typically CJK)
>>> and combining marks. So UTF-32 buys you absolutely nothing compared
>>> to UTF-8.
>> <
>> Why would you want zero-width spaces ?
>
> They show the boundaries between words in scripts that don't use
> explicit spaces, and they're useful in Latin text to show places where
> you can break a long word. There are also Zero Width Joiner and Zero
> Width Nonjoiner codes which are useful in languages like Arabic where
> adjacent characters are sometimes combined except when they aren't.
>

They can also used to break up kerning pairs. In some text rendering
systems, letter combinations such as AV are placed a little closer
together, with the top left of the V above the bottom right of the A. A
zero-width space can break that, stopping the overlap.

> Pretty much anything you think you know about text layout is wrong
> in at least some of the languages people use in computer text.
>

I find that people get a many things wrong, or at least imprecise, even
if they stick to the one language.

(The TeXbook should come with a warning label - reading it teaches you
huge amounts about typesetting, but leaves you unable to look at other
people's writing without getting annoyed about their poor layout or
inconsistencies!)