Hi,

I use a library to format and print debugging message which use also Linux-terminal color-code.

the CORE problem is that "string length .." is used to format the message and add additional information etc
BUT the "string length .." count the NON-visible color-code as well as visible chars that result in a MISS-format string

example: (color is not visible in this thread !! - but all is "green")

-> outTXT<multi-line>
-> | |DEBUG[1]: aa -> LIST | <- count color codes
-> | |DEBUG[1]: -> | a1 : 111 | <- count color codes
-> | |DEBUG[1]: -> | a2 : 222 | <- count color codes
-> | |DEBUG[1]: -> | a3 : 333 |

a color-code is something like

## ------------------------------------------------------------------------
## DEBUG helper
## Black 0;30 Dark Gray 1;30
## Red 0;31 Light Red 1;31
## Green 0;32 Light Green 1;32
## Brown/Orange 0;33 Yellow 1;33
## Blue 0;34 Light Blue 1;34
## Purple 0;35 Light Purple 1;35
## Cyan 0;36 Light Cyan 1;36
## Light Gray 0;37 White 1;37
##
## 8bit (256) colors: https://stackoverflow.com/questions/4842424/list-of-ansi-color-escape-sequences
variable CL_COLOR
set CL_COLOR(red) "\[1;31m"
set CL_COLOR(green) "\[1;32m"
set CL_COLOR(yellow) "\[1;33m"
set CL_COLOR(blue) "\[1;34m"
set CL_COLOR(purple) "\[38;5;206m"
set CL_COLOR(cyan) "\[1;36m"
set CL_COLOR(lightcyan) "\[38;5;51m"
set CL_COLOR(white) "\[1;37m"
set CL_COLOR(grey) "\[38;5;254m"
set CL_COLOR(orange) "\[38;5;202m"
set CL_COLOR(no) ""
variable CL_RESET "\[0;m"

code:

set tstmsg "$lib_debug::CL_COLOR(red)123$lib_debug::CL_RESET"

puts "$tstmsg"
puts [string length $tstmsg]

result:

123 (the color of "123" is red)
15

there is something MISSING in TCL and this is the "string length ..." only count the !! VISIBLE !! chars

mfg ao

→ add the problem to TCL-wiki also: https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars

aotto1968 <aotto1968@t-online.de> wrote:
> there is something MISSING in TCL and this is the "string length ..."
> only count the !! VISIBLE !! chars

No. The solution is to measure the string first, then add the terminal
escapes in a lower level, just before output to the terminal.

There is no need to complicate string length with such a requirement,
as it is has not ever been defined as measuring only 'visible'
characters. It has always measured "all" characters in the string.
And it should continue to do so.

All those terminal escapes are simply "characters in the string" which
"increase its length". They only become "invisible" when the string is
output to a device that interprets them in that way. Output them to a
disk file and they appear as bytes in the file.

Secondly, what is "visible" varies depending upon what terminal is used
for output, so making this work for all cases is a deep rabbit hole
with potentially no bottom.

On 3/5/24 09:19, aotto1968 wrote:
>
> → add the problem to TCL-wiki also:
> https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars
> Noted in said WIKI page that what characters are visible is *VERY*
device dependent and that this is not something TCL is missing but
rather the application.

This is funny → it seems to be a "serious" problem to "format" the "terminal-output" with
invisible ctrl-character(s)

I understand that at least a TERM environment variable have to be available to understand
what is a ctrl-char and what is a printing-char.

On 05.03.24 16:21, Rich wrote:
> aotto1968 <aotto1968@t-online.de> wrote:
>> there is something MISSING in TCL and this is the "string length ..."
>> only count the !! VISIBLE !! chars
>
> No. The solution is to measure the string first, then add the terminal
> escapes in a lower level, just before output to the terminal.
>
> There is no need to complicate string length with such a requirement,
> as it is has not ever been defined as measuring only 'visible'
> characters. It has always measured "all" characters in the string.
> And it should continue to do so.
>
> All those terminal escapes are simply "characters in the string" which
> "increase its length". They only become "invisible" when the string is
> output to a device that interprets them in that way. Output them to a
> disk file and they appear as bytes in the file.
>
> Secondly, what is "visible" varies depending upon what terminal is used
> for output, so making this work for all cases is a deep rabbit hole
> with potentially no bottom.

On 3/5/24 13:02, aotto1968 wrote:
>
> This is funny → it seems to be a "serious" problem to "format" the
> "terminal-output" with
> invisible ctrl-character(s)
>
> I understand that at least a TERM environment variable have to be
> available to understand
> what is a ctrl-char and what is a printing-char.

You are making the invalid assumption that all strings only are output
to a terminal -- for most application few if any strings are output to a
terminal. Many times string are just used to send to a socket to
communicate with clients/servers.

>
>
> On 05.03.24 16:21, Rich wrote:
>> aotto1968 <aotto1968@t-online.de> wrote:
>>> there is something MISSING in TCL and this is the "string length ..."
>>> only count the !!  VISIBLE !!  chars
>>
>> No.  The solution is to measure the string first, then add the terminal
>> escapes in a lower level, just before output to the terminal.
>>
>> There is no need to complicate string length with such a requirement,
>> as it is has not ever been defined as measuring only 'visible'
>> characters.  It has always measured "all" characters in the string.
>> And it should continue to do so.
>>
>> All those terminal escapes are simply "characters in the string" which
>> "increase its length".  They only become "invisible" when the string is
>> output to a device that interprets them in that way.  Output them to a
>> disk file and they appear as bytes in the file.
>>
>> Secondly, what is "visible" varies depending upon what terminal is used
>> for output, so making this work for all cases is a deep rabbit hole
>> with potentially no bottom.
>

On 3/5/2024 2:02 PM, aotto1968 wrote:
>
> This is funny → it seems to be a "serious" problem to "format" the
> "terminal-output" with
> invisible ctrl-character(s)
>
> I understand that at least a TERM environment variable have to be
> available to understand
> what is a ctrl-char and what is a printing-char.
>
>

[string length] doesn't know about your intentions around how to handle
strings you pass to it. If you want to skip color codes, you can easily
modify your input. string-map is your friend here:

% string length [string map [list $lib_debug::CL_COLOR(red) {}
$lib_debug::CL_COLOR(green) {}
...
$lib_debug::CL_RESET {}] $tstmsg]

This should give you the results you are after.

saitology9 <saitology9@gmail.com> wrote:
> On 3/5/2024 2:02 PM, aotto1968 wrote:
>>
>> This is funny → it seems to be a "serious" problem to "format" the
>> "terminal-output" with
>> invisible ctrl-character(s)
>>
>> I understand that at least a TERM environment variable have to be
>> available to understand
>> what is a ctrl-char and what is a printing-char.
>>
>>
>
> [string length] doesn't know about your intentions around how to handle
> strings you pass to it. If you want to skip color codes, you can easily
> modify your input. string-map is your friend here:
>
> % string length [string map [list $lib_debug::CL_COLOR(red) {}
> $lib_debug::CL_COLOR(green) {}
> ...
> $lib_debug::CL_RESET {}] $tstmsg]
>
> This should give you the results you are after.

Or, the OP could measure the string *before* they wrap it with terminal
control characters (which *do* make the string longer) for output to a
terminal.

aotto1968 <aotto1968@t-online.de> wrote:
> On 05.03.24 16:21, Rich wrote:
>> aotto1968 <aotto1968@t-online.de> wrote:
>>> there is something MISSING in TCL and this is the "string length
>>> ..." only count the !! VISIBLE !! chars
>>
>> No. The solution is to measure the string first, then add the
>> terminal escapes in a lower level, just before output to the
>> terminal.
>>
>> All those terminal escapes are simply "characters in the string" which
>> "increase its length". They only become "invisible" when the string is
>> output to a device that interprets them in that way. Output them to a
>> disk file and they appear as bytes in the file.
>>
>> Secondly, what is "visible" varies depending upon what terminal is used
>> for output, so making this work for all cases is a deep rabbit hole
>> with potentially no bottom.
>
> This is funny → it seems to be a "serious" problem to "format" the
> "terminal-output" with invisible ctrl-character(s)
>
> I understand that at least a TERM environment variable have to be
> available to understand what is a ctrl-char and what is a
> printing-char.

There is no way for Tcl to know if you plan to send that string to the
terminal defined by the TERM variable.

You might do that. Or you might send it to a file, or your stdout
might be redirected to a file, or you might send it to a socket and
onward to another machine, where it might be output to a very different
'terminal' from the one where it was created.

If you want string lengths of strings with terminal control sequences
ignored, then simply measure your strings *before* you wrap them with
terminal control sequences.

I understand the the default `string length ...` is the wrong place to add this feature
*but* there could be other options.

1. string -visible length "..." or "string -terminal length"
2. or add a "terminal command for all "terminal" related functions like color etc
→ terminal string length "..."
→ terminal color ...
3. your already have such a thing like `string is control ..` or `string is ...` character classes

and yes you can always do a massive code-reorganization to get a simple `string length …` working
without invisible character chars count.

a separate C-library for this task would be good because all other languages seems to have the same "problem"
using the "terminal" proper.

On 05.03.24 21:42, Rich wrote:
> aotto1968 <aotto1968@t-online.de> wrote:
>> On 05.03.24 16:21, Rich wrote:
>>> aotto1968 <aotto1968@t-online.de> wrote:
>>>> there is something MISSING in TCL and this is the "string length
>>>> ..." only count the !! VISIBLE !! chars
>>>
>>> No. The solution is to measure the string first, then add the
>>> terminal escapes in a lower level, just before output to the
>>> terminal.
>>>
>>> All those terminal escapes are simply "characters in the string" which
>>> "increase its length". They only become "invisible" when the string is
>>> output to a device that interprets them in that way. Output them to a
>>> disk file and they appear as bytes in the file.
>>>
>>> Secondly, what is "visible" varies depending upon what terminal is used
>>> for output, so making this work for all cases is a deep rabbit hole
>>> with potentially no bottom.
>>
>> This is funny → it seems to be a "serious" problem to "format" the
>> "terminal-output" with invisible ctrl-character(s)
>>
>> I understand that at least a TERM environment variable have to be
>> available to understand what is a ctrl-char and what is a
>> printing-char.
>
> There is no way for Tcl to know if you plan to send that string to the
> terminal defined by the TERM variable.
>
> You might do that. Or you might send it to a file, or your stdout
> might be redirected to a file, or you might send it to a socket and
> onward to another machine, where it might be output to a very different
> 'terminal' from the one where it was created.
>
> If you want string lengths of strings with terminal control sequences
> ignored, then simply measure your strings *before* you wrap them with
> terminal control sequences.
>

In article <us9epm$amac$1@dont-email.me>,
aotto1968 <aotto1968@t-online.de> wrote:
>
>I understand the the default `string length ...` is the wrong place to
>add this feature
>*but* there could be other options.
>
>1. string -visible length "..." or "string -terminal length"
>2. or add a "terminal command for all "terminal" related functions like
>color etc
> → terminal string length "..."
> → terminal color ...
>3. your already have such a thing like `string is control ..` or `string
>is ...` character classes
>
>and yes you can always do a massive code-reorganization to get a simple
>`string length …` working
>without invisible character chars count.
>
>a separate C-library for this task would be good because all other
>languages seems to have the same "problem"
>using the "terminal" proper.
>

There is such a C library. It's called "curses"

It appears there is a tcllib package "term" which does similar things,
though I have never tried it.
--
columbiaclosings.com
What's not in Columbia anymore..

On Tue, 5 Mar 2024 16:03:45 +0100
aotto1968 <aotto1968@t-online.de> wrote:

>
> Hi,
>
> I use a library to format and print debugging message which use also Linux-terminal color-code.
>
> the CORE problem is that "string length .." is used to format the message and add additional information etc
> BUT the "string length .." count the NON-visible color-code as well as visible chars that result in a MISS-format string

A simple helper proc is enough:

proc termlen {str} {
string length [regsub -all {\u001b\[[0-9;]+m} $str {}]
}
set CL_COLOR(red) "\u001b\[1;31m"
set CL_RESET "\u001b\[0;m"

set msg "This is a test!!"
set tstmsg $CL_COLOR(red)$msg$CL_RESET
puts $tstmsg

puts "msg length : [string length $msg]"
puts "tstmsg length : [string length $tstmsg]"
puts "termlen : [termlen $tstmsg]"

Output:

This is a test!! (output in red)
msg length : 16
tstmsg length : 28
termlen : 16

Also, be sure that the channel you are writing to is a terminal and no
a regular file; on *nix terminals at least this check works (channels
have a -mode option when the C function isatty(fd) returns true).

proc isterminal {chan} {expr {![catch {chan configure $chan -mode}]}}

~$ tclsh 2>/dev/null
% proc isterminal {chan} {expr {![catch {chan configure $chan -mode}]}}
% isterminal stdout
1 % isterminal stderr
0

--
Emiliano

On 3/6/24 03:59, aotto1968 wrote:
>
> I understand the the default `string length ...` is the wrong place to
> add this feature
> *but* there could be other options.
>
> 1. string -visible length "..." or "string -terminal length"
> 2. or add a "terminal command for all "terminal" related functions like
> color etc
>    → terminal string length "..."
>    → terminal color ...
> 3. your already have such a thing like `string is control ..` or `string
> is ...` character classes
>
> and yes you can always do a massive code-reorganization to get a simple
> `string length …` working
> without invisible character chars count.
>
> a separate C-library for this task would be good because all other
> languages seems to have the same "problem"
> using the "terminal" proper.

Please feel free to code up and contribute, via a TIP, your suggestions.

>
> On 05.03.24 21:42, Rich wrote:
>> aotto1968 <aotto1968@t-online.de> wrote:
>>> On 05.03.24 16:21, Rich wrote:
>>>> aotto1968 <aotto1968@t-online.de> wrote:
>>>>> there is something MISSING in TCL and this is the "string length
>>>>> ..." only count the !!  VISIBLE !!  chars
>>>>
>>>> No.  The solution is to measure the string first, then add the
>>>> terminal escapes in a lower level, just before output to the
>>>> terminal.
>>>>
>>>> All those terminal escapes are simply "characters in the string" which
>>>> "increase its length".  They only become "invisible" when the string is
>>>> output to a device that interprets them in that way.  Output them to a
>>>> disk file and they appear as bytes in the file.
>>>>
>>>> Secondly, what is "visible" varies depending upon what terminal is used
>>>> for output, so making this work for all cases is a deep rabbit hole
>>>> with potentially no bottom.
>>>
>>> This is funny → it seems to be a "serious" problem to "format" the
>>> "terminal-output" with invisible ctrl-character(s)
>>>
>>> I understand that at least a TERM environment variable have to be
>>> available to understand what is a ctrl-char and what is a
>>> printing-char.
>>
>> There is no way for Tcl to know if you plan to send that string to the
>> terminal defined by the TERM variable.
>>
>> You might do that.  Or you might send it to a file, or your stdout
>> might be redirected to a file, or you might send it to a socket and
>> onward to another machine, where it might be output to a very different
>> 'terminal' from the one where it was created.
>>
>> If you want string lengths of strings with terminal control sequences
>> ignored, then simply measure your strings *before* you wrap them with
>> terminal control sequences.
>>
>

This thread reminded me of my movie subtitle program.

Subtitles have a limit on number of characters per line. My program
has to count characters all the time and enforce the limit. And it
turns out that some subtitle formats provide for phrases to be possibly
rendered in italics with run-of-the-mill <i>HTML formatting</i>.

Of course, the HTML tags are not visible on the screen so they must
not be counted.

So my code reads the line into a separate variable, strips the HTML
code and counts the characters in that separate variable.

Seems to be pretty much the same case at hand here.

--
Luc
>>

I update the wiki: https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars

update 7 mar 2024
The CORE problem is not just the string length the CORE problem is the representation of the string in tcl

(and all other programming languages as well)
Let's start simple: the problem is just the same as the multibyte-chars-problem raised up ~30 years ago.

at the beginning it was only ascii and a string was char* after a while they figure out that are languages with more chars than
plain ascii.
It took ~20 years to implement utf8 and finally utf16 into all programming languages.
tcl uses utf8 as script-encoding and utf16 internal as string representation because

a string is only useful if index operations can be performt like: string index $str 0 and utf16 is index-able and utf8 not.
The string length problem is not alone, there is also the index-operation-problem like:

string index or
string range or
format %40s
that doesn't work.

The CORE problem is that the terminal-chars are embedded into the string and not as an character attribute.

* the terminal is the main presentation-layer of a tcl. * the gui-toolkit is the main presentation-layer of tk.

The current string length in tcl is doing something in-between string bytelength and string visual-length because
multibyte-chars are reduced to a single-char but invisible console-ctrl-char are counted like a char.

solution
The solution is the same solution like in utf16 or tk. every char have to be presented by a data-structure that hold

the multibyte-char (utf16)
the terminal-encoding (color,bold,underline,...)
I call this utf32

The puts operate as:

connected to a terminal just translate the utf32 string into a stream understood by the terminal.
connected to a file just do a utf32 write into a file
(perhaps the utf32 can be encoded into utf8)
finally you can write a terminal-colored-string into a file and read it again with the color information still available.
I understand that this is not a job for tcl alone because all programming languages are involved and that is the reason I assume
a utf32 consortium is good and programmers of all languages start together to implement this shit.

On 05.03.24 16:03, aotto1968 wrote:
>
> Hi,
>
> I use a library to format and print debugging message which use also Linux-terminal color-code.
>
> the CORE problem is that "string length .." is used to format the message and add additional information etc
> BUT the "string length .." count the NON-visible color-code as well as visible chars that result in a MISS-format string
>
> example: (color is not visible in this thread !! - but all is "green")
>
>  -> outTXT<multi-line>
>  ->   |   |DEBUG[1]: aa   -> LIST                                     |  <- count color codes
>  ->   |   |DEBUG[1]:      ->   | a1                   : 111           |  <- count color codes
>  ->   |   |DEBUG[1]:      ->   | a2                   : 222           |  <- count color codes
>  ->   |   |DEBUG[1]:      ->   | a3                   : 333 |
>
>
> a color-code is something like
>
>   ## ------------------------------------------------------------------------
>   ## DEBUG helper
>   ##  Black        0;30     Dark Gray     1;30
>   ##  Red          0;31     Light Red     1;31
>   ##  Green        0;32     Light Green   1;32
>   ##  Brown/Orange 0;33     Yellow        1;33
>   ##  Blue         0;34     Light Blue    1;34
>   ##  Purple       0;35     Light Purple  1;35
>   ##  Cyan         0;36     Light Cyan    1;36
>   ##  Light Gray   0;37     White         1;37
>   ##
>   ##  8bit (256) colors: https://stackoverflow.com/questions/4842424/list-of-ansi-color-escape-sequences
>   variable CL_COLOR
>   set  CL_COLOR(red)        "\[1;31m"
>   set  CL_COLOR(green)      "\[1;32m"
>   set  CL_COLOR(yellow)     "\[1;33m"
>   set  CL_COLOR(blue)       "\[1;34m"
>   set  CL_COLOR(purple)     "\[38;5;206m"
>   set  CL_COLOR(cyan)       "\[1;36m"
>   set  CL_COLOR(lightcyan)  "\[38;5;51m"
>   set  CL_COLOR(white)      "\[1;37m"
>   set  CL_COLOR(grey)       "\[38;5;254m"
>   set  CL_COLOR(orange)     "\[38;5;202m"
>   set  CL_COLOR(no)         ""
>   variable  CL_RESET        "\[0;m"
>
>
> code:
>
> set tstmsg  "$lib_debug::CL_COLOR(red)123$lib_debug::CL_RESET"
>
> puts "$tstmsg"
> puts [string length $tstmsg]
>
>
> result:
>
> 123        (the color of "123" is red)
> 15
>
>
> there is something MISSING in TCL and this is the "string length ..." only count the !! VISIBLE !! chars
>
>
> mfg ao

In article <usc1bu$vli2$1@dont-email.me>,
aotto1968 <aotto1968@t-online.de> wrote:
>I update the wiki:
>https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars
>
>update 7 mar 2024
>The CORE problem is not just the string length the CORE problem is the
>representation of the string in tcl
>
>(and all other programming languages as well)
>Let's start simple: the problem is just the same as the
>multibyte-chars-problem raised up ~30 years ago.
>
>at the beginning it was only ascii and a string was char* after a while
>they figure out that are languages with more chars than
>plain ascii.
>It took ~20 years to implement utf8 and finally utf16 into all
>programming languages.
>tcl uses utf8 as script-encoding and utf16 internal as string
>representation because
>
>a string is only useful if index operations can be performt like: string
>index $str 0 and utf16 is index-able and utf8 not.
>The string length problem is not alone, there is also the
>index-operation-problem like:
>
>string index or
>string range or
>format %40s
>that doesn't work.
>
>The CORE problem is that the terminal-chars are embedded into the string
>and not as an character attribute.
>
>* the terminal is the main presentation-layer of a tcl. * the
>gui-toolkit is the main presentation-layer of tk.
>
the terminal is the main presentation-layer of a tcl

But it's not. At this point, I would guess sockets are the main
"presentation layer", certainly the REST work I have been doing over
the past few years does no terminal I/O. If you want to talk to a
terminal use curses or whatever. That has no relation to the use
cases for files & sockets.
--
columbiaclosings.com
What's not in Columbia anymore..

If you interact with tcl, read/write debugging messages etc it is always a terminal or a gui application.
the "socket" is just like the "file" an other target. I understand that sending "additional-terminal-attributes"
over the socket is *not* a all-day behavior because:

1. you don't use *tcl* for that
2. there is already a *ssh* application
3. there are format(s) like *html* etc which add attributes to strings.

the "terminal-library" as you mentioned does a much but mostly not basic "string-index" operations
because this is the target of the programming language.

→ try to find a [string range $str 17 45] in a curses library etc.

On 07.03.24 14:51, Ted Nolan <tednolan> wrote:
> In article <usc1bu$vli2$1@dont-email.me>,
> aotto1968 <aotto1968@t-online.de> wrote:
>> I update the wiki:
>> https://wiki.tcl-lang.org/page/BUG+%2D+%27string+length%27+count+also+NON+visible+chars
>>
>> update 7 mar 2024
>> The CORE problem is not just the string length the CORE problem is the
>> representation of the string in tcl
>>
>> (and all other programming languages as well)
>> Let's start simple: the problem is just the same as the
>> multibyte-chars-problem raised up ~30 years ago.
>>
>> at the beginning it was only ascii and a string was char* after a while
>> they figure out that are languages with more chars than
>> plain ascii.
>> It took ~20 years to implement utf8 and finally utf16 into all
>> programming languages.
>> tcl uses utf8 as script-encoding and utf16 internal as string
>> representation because
>>
>> a string is only useful if index operations can be performt like: string
>> index $str 0 and utf16 is index-able and utf8 not.
>> The string length problem is not alone, there is also the
>> index-operation-problem like:
>>
>> string index or
>> string range or
>> format %40s
>> that doesn't work.
>>
>> The CORE problem is that the terminal-chars are embedded into the string
>> and not as an character attribute.
>>
>> * the terminal is the main presentation-layer of a tcl. * the
>> gui-toolkit is the main presentation-layer of tk.
>>
> the terminal is the main presentation-layer of a tcl
>
> But it's not. At this point, I would guess sockets are the main
> "presentation layer", certainly the REST work I have been doing over
> the past few years does no terminal I/O. If you want to talk to a
> terminal use curses or whatever. That has no relation to the use
> cases for files & sockets.