On 1/2/2024 8:00 AM, Rich wrote:
> Luc <luc@sep.invalid> wrote:
>> I had another idea. I want to create the slow widgets in background
>> auxiliary threads and keep them hidden (unpacked) until they're needed.
>>
>> After toying with threads and this idea for some time it seems it works,
>> I've accomplished everything except that I can't see any way to bring
>> the widget from the auxiliary thread to the main thread and application.
>>
>> Is there a way?
>
> No, not to transfer a Tk widget from one thread to another.
>
> Tcl threads are share nothing threads (each interperter is isolated
> from the others), and Tk widgets are created in a particuar interpreter
> and can only exist in the interpreter they were created in.
>
> The most you can do is have the background thread periodically do a
> 'thread::send' to the thread where the widget exists to do a small bit
> of the work at a time.
>
> Here's your demo code modified to use a 'threads' mentod for filling the
> text widget:
>
> #!/usr/bin/tclsh
>
> package require Tk
> package require Thread
>
> set sb [scrollbar .sb -orient vertical -command [list .tx yview]]
> set tx [text .tx -yscrollcommand [list $sb set]]
> pack $tx -side left -fill both -expand 1
> pack $sb -fill y -side right
> focus $tx
> bind . <Escape> [list exit 0]
>
> font create myfont -family Freesans -size 14
> image create photo IMAGE -file /usr/share/icons/hicolor/32x32/apps/kicad.png
>
> set begin [clock milliseconds]
> set end [clock milliseconds]
> puts "[expr {$end - $begin}] ms elapsed in main interpreter"
>
> set thread [thread::create -preserved]
>
> thread::send $thread [list set parent [thread::id]]
>
> thread::send -async $thread {
> set begin [clock milliseconds]
> for {set x 1} {$x <= 2000} {incr x} {
> thread::send $parent [subst -nocommands {
> set b [button .b$x -font myfont -text $x -width 80 -cursor arrow -image IMAGE -compound left]
> \$tx window create end -window \$b
> \$tx insert end "\n"
> }]
> after 100
> }
> set end [clock milliseconds]
> puts "[expr {$end - $begin}] ms elapsed in thread interpreter"
> thread::send -async $parent {thread::release $thread}
> }
>
> Note that it inserts a new row every 100ms, so that it is slow enough
> to "see" the insertions happening. Reduce the wait in the 'after 100'
> to speed things up. You can also remove the after wait for the
> "fastest possible" fill. With no after wait at all the thread takes
> 7388ms here to finish filling the text.

Wouldn't it be far simpler to just use a proc with an after that keeps propagating itself:

proc do_a_widget args {
.... build this widget ...
after 100 do_a_widget ... ;# next widget
} after 0 do_a_widget ...

The only work the separate thread is doing is a for loop.

However, it seems that if the buttons are independent of the text widget, i.e. they don't have to be in the same toplevel or frame etc. then it would be possible to build the buttons in one thread, and leave them there. Then any action, i.e. button press -command would simply do a thread::send back to main which would then handle the button push. Likewise, any button config would have to be a thread::send to the thread with the buttons. The difficulty could be synchronizing the positions of the buttons vs the text widget. Think of it as having 2 windows along side each other.

However, only Luc can determine if all this extra bookkeeping is worth the hassle. It seems that a better approach would be to only generate the buttons on demand. Maybe have a proc that takes a start button number and a count of buttons and will display them - and build any new ones not already built. How long could it take to create a screen-full of them?

One caveat with Tk and threads however, prior to 8.6.12, linux tcl would crash occasionally when working with Tk in more than one thread. I filed a ticket on that some years back and Christian added some locks. I haven't really verified that it fixed the problem. Note that this was a linux specific problem, I use Tk in multiple threads in windows without any problems.

et99 <et99@rocketship1.me> wrote:
>
> Wouldn't it be far simpler to just use a proc with an after that keeps propagating itself:

Yes, but Luc implied he wanted a 'threads' version.

> proc do_a_widget args {
> ... build this widget ...
> after 100 do_a_widget ... ;# next widget
> }
> after 0 do_a_widget ...
>
> The only work the separate thread is doing is a for loop.

Yep, same end result, different path to get there.

> However, it seems that if the buttons are independent of the text
> widget, i.e. they don't have to be in the same toplevel or frame
> etc. then it would be possible to build the buttons in one thread,
> and leave them there.

The buttons are only present because Luc discovered that embedding an
image in a button, then inserting the button into the text, was
significantly faster than a native 'image create' to place the same
image in the same spot in the text. I.e, the buttons are a workaround
for a very slow bit of code in the "image create" code path in the text
widget.

> Then any action, i.e. button press -command would simply do a
> thread::send back to main which would then handle the button push.
> Likewise, any button config would have to be a thread::send to the
> thread with the buttons. The difficulty could be synchronizing the
> positions of the buttons vs the text widget. Think of it as having 2
> windows along side each other.

It would have to be, because the buttons in the second thread can't be
packed or inserted into anything in the first thread.

> However, only Luc can determine if all this extra bookkeeping is
> worth the hassle. It seems that a better approach would be to only
> generate the buttons on demand. Maybe have a proc that takes a start
> button number and a count of buttons and will display them - and
> build any new ones not already built. How long could it take to
> create a screen-full of them?

Just a screen full is fast (well, fast enough to not worry about the
time it takes). Creating 12,500 of them is where things slowed down
for Luc and this message thread was begun.

Greetings from the drawing board.

Thank you for the additional ideas. I really dislike the idea with
buttons though. I think it's awkward, ugly, difficult to work with and
it flickers a lot when moving.

It's not true that buttons are faster. They were with a default font,
but that is not acceptable. The user *will* be changing fonts, and that
makes buttons just as slow as a text widget. That idea is scrapped.

Rich posted a very good solution yesterday. I adapted it and did it
with a text widget instead of buttons. The prototype works very well.
I made it play sounds while the text widget is filled up with text and
images in the background thread. And the GUI was very responsive during
the entire operation. A success.

But this is me we are talking about and I messed that up too. When I
ported the idea to the real application, it works, but everything
freezes completely until the task is done. I must be doing something
wrong again of course. Now I need to investigate and/or try other
designs until the concept works as expected.

I will keep you posted.

--
Luc
>>

I am having a lot of frustration with the scope of variables.

Rich posted something like this:

thread::send -async $thread {
set begin [clock milliseconds]
for {set x 1} {$x <= 8500} {incr x} {
thread::send $parent [subst -nocommands {
\$tx insert end "$x: "
\$tx image create end -image IMAGE
\$tx insert end ":$x\n"
}]
}
set end [clock milliseconds]
puts "[expr {$end - $begin}] ms elapsed in thread interpreter"
thread::send -async $parent {thread::release $thread}
}

Escaping the variables with \$ only works with globals. I have the
thread running inside a proc. Man, I can't find a way to see the
thread see the variables within the scope of the proc.

Things begin to work when I use globals, but that will be cluttering
the global namespace too much during real usage. Even tsv is not
working. Is there a way?

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> But this is me we are talking about and I messed that up too. When I
> ported the idea to the real application, it works, but everything
> freezes completely until the task is done. I must be doing something
> wrong again of course.

Most likely you omitted an -async somewhere.

Luc <luc@sep.invalid> wrote:
> I am having a lot of frustration with the scope of variables.
>
> Rich posted something like this:
>
> thread::send -async $thread {
> set begin [clock milliseconds]
> for {set x 1} {$x <= 8500} {incr x} {
> thread::send $parent [subst -nocommands {
> \$tx insert end "$x: "
> \$tx image create end -image IMAGE
> \$tx insert end ":$x\n"
> }]
> }
> set end [clock milliseconds]
> puts "[expr {$end - $begin}] ms elapsed in thread interpreter"
> thread::send -async $parent {thread::release $thread}
> }
>
>
> Escaping the variables with \$ only works with globals.

Yes, this was a 'quick and dirty example'. And globals was simply the
easiest way to go for a 'quick and dirty example'.

> I have the thread running inside a proc. Man, I can't find a way to
> see the thread see the variables within the scope of the proc.
>
> Things begin to work when I use globals, but that will be cluttering
> the global namespace too much during real usage. Even tsv is not
> working. Is there a way?

Yes, setup a proc in the GUI thread that will be called to do the work
of "adding" things, one at a time, to the text widget.

Then repeatadly call the proc (either with an after loop, or from a
background thread). If all you are doing is repeatadly calling the
proc via the event loop, an after loop will be in general simpler.

On Wed, 3 Jan 2024 20:32:52 -0000 (UTC), Rich wrote:

>Yes, setup a proc in the GUI thread that will be called to do the work
>of "adding" things, one at a time, to the text widget.
>
>Then repeatadly call the proc (either with an after loop, or from a
>background thread). If all you are doing is repeatadly calling the
>proc via the event loop, an after loop will be in general simpler.

I've read this multiple times and can't understand what you mean.

Either way, I've made tsv to work and everything is working now...

except that the application still freezes while the task is running.
I use -async in all thread::send commands and it's still not working as
it should. And I basically copied everything from the prototype that
works, just chaning the specific/relevant commands.

Sigh. I'm stumped again. :-(

--
Luc
>>

On 1/3/2024 1:37 PM, Luc wrote:
> On Wed, 3 Jan 2024 20:32:52 -0000 (UTC), Rich wrote:
>
>> Yes, setup a proc in the GUI thread that will be called to do the work
>> of "adding" things, one at a time, to the text widget.
>>
>> Then repeatadly call the proc (either with an after loop, or from a
>> background thread). If all you are doing is repeatadly calling the
>> proc via the event loop, an after loop will be in general simpler.
>
> I've read this multiple times and can't understand what you mean.
>
> Either way, I've made tsv to work and everything is working now...
>
> except that the application still freezes while the task is running.
> I use -async in all thread::send commands and it's still not working as
> it should. And I basically copied everything from the prototype that
> works, just chaning the specific/relevant commands.
>
> Sigh. I'm stumped again. :-(
>
Are you sending from the alternate thread to the main thread an [after] command?

From the manual:

after ms
Ms must be an integer giving a time in milliseconds. The command sleeps for ms milliseconds and then returns. While the command is sleeping the application does not respond to events.

Luc <luc@sep.invalid> wrote:
> On Wed, 3 Jan 2024 20:32:52 -0000 (UTC), Rich wrote:
>
>>Yes, setup a proc in the GUI thread that will be called to do the work
>>of "adding" things, one at a time, to the text widget.
>>
>>Then repeatadly call the proc (either with an after loop, or from a
>>background thread). If all you are doing is repeatadly calling the
>>proc via the event loop, an after loop will be in general simpler.
>
> I've read this multiple times and can't understand what you mean.

The reason the quick and dirty threads example does not freeze the UI
is because the background thread is doing this:

while X<12500 do
call-main-thread-add-one-line
pause-for-a-tiny-bit
done

I.e., it "repeatedly" call's "the proc" (in this case it called a
"script") with a small pause between each call.

The script only did one thing, it added one single line to the text
widget. Then it (the script) was done, and the script exited.

So you setup your "proc" to do only one small part of the whole job.

Then you arrange to "run the proc" over and over, each run doing one
small part. That "run it over and over" can be done by having a thread
call the proc, or by using an after calling loop.

An 'after calling loop' looks like this:

# the proc, which does "one thing" (assume a text widget named .text
# exists and is already packed/gridded/placed
proc do-one-thing {thing} {
.text insert end $thing
# check if we are done doing things, stop if so:
if {$thing >= 12500} { return }
# setup so the event loop will call "do-one-thing" again
after idle [list do-one-thing [incr thing]]
}

# start the "after loop"
after 100 [list do-one-thing 0]

# and enter the event loop -- if in Tk, just return from any proc, or
# run off the end of the script. If just Tcl, one has to do a
# "vwait forever" to start up the event loop. Note that "forever"
# can be any variable name you like.

> except that the application still freezes while the task is running.
> I use -async in all thread::send commands and it's still not working
> as it should. And I basically copied everything from the prototype
> that works, just chaning the specific/relevant commands.
>
> Sigh. I'm stumped again. :-(

Somewhere in the app you are blocking the event loop from running.
You'll have to hunt that down and fix it. Most likely you were already
blocking the event loop on /something/ before you copied in the thread
example, but you didn't notice you were blocking it because you
expected the app to freeze while the widget was loaded.

On Wed, 3 Jan 2024 23:40:24 -0000 (UTC), Rich wrote:

>> Sigh. I'm stumped again. :-(
>
>Somewhere in the app you are blocking the event loop from running.
>You'll have to hunt that down and fix it. Most likely you were already
>blocking the event loop on /something/ before you copied in the thread
>example, but you didn't notice you were blocking it because you
>expected the app to freeze while the widget was loaded.

Can you give me one or two examples of how I could possibly be blocking
the event loop? I honestly don't know what to look for.

Another question: like I said, you posted something like this:

thread::send -async $thread {
set begin [clock milliseconds]
for {set x 1} {$x <= 8500} {incr x} {
thread::send $parent [subst -nocommands {
\$tx insert end "$x: "
\$tx image create end -image IMAGE
\$tx insert end ":$x\n"
}]
}
set end [clock milliseconds]
puts "[expr {$end - $begin}] ms elapsed in thread interpreter"
thread::send -async $parent {thread::release $thread}
}

It works fine in the global scope.

When I put that in a proc, the last line throws an error saying that
there is no $thread variable. How do I make that work correctly inside
a proc?

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> On Wed, 3 Jan 2024 23:40:24 -0000 (UTC), Rich wrote:
>
>>> Sigh. I'm stumped again. :-(
>>
>>Somewhere in the app you are blocking the event loop from running.
>>You'll have to hunt that down and fix it. Most likely you were already
>>blocking the event loop on /something/ before you copied in the thread
>>example, but you didn't notice you were blocking it because you
>>expected the app to freeze while the widget was loaded.
>
> Can you give me one or two examples of how I could possibly be blocking
> the event loop? I honestly don't know what to look for.

Simple, anytime any of your Tcl code is running, the event loop is
blocked.

> Another question: like I said, you posted something like this:
>
> thread::send -async $thread {
> set begin [clock milliseconds]
> for {set x 1} {$x <= 8500} {incr x} {
> thread::send $parent [subst -nocommands {
> \$tx insert end "$x: "
> \$tx image create end -image IMAGE
> \$tx insert end ":$x\n"
> }]
> }
> set end [clock milliseconds]
> puts "[expr {$end - $begin}] ms elapsed in thread interpreter"
> thread::send -async $parent {thread::release $thread}
> }
>
>
> It works fine in the global scope.
>
> When I put that in a proc, the last line throws an error saying that
> there is no $thread variable. How do I make that work correctly inside
> a proc?

When you say "put that in a proc" what do you mean. Do you mean
literallly wrapping "proc something {args} { ... } around the code
above verbatim?

If so, then go back and look at what I posted and note that if you
wrapped "the above snippet" in a proc, that you overlooked or ignored
the line above thread::send where the "thread" variable was defined.
It holds the thread id of the created child thread (and you need a
thread's id value to be able to "send" anything to it. You also
probably overlooked the other line where a thread send was used to set
a variable 'parent' in the child thread telling it the thread id value
of its parent.

On Thu, 4 Jan 2024 06:35:18 -0000 (UTC), Rich wrote:

>> Can you give me one or two examples of how I could possibly be blocking
>> the event loop? I honestly don't know what to look for.
>
>Simple, anytime any of your Tcl code is running, the event loop is
>blocked.

I had no clue what you meant by that but I spent a long time rebuilding
a lot of the app brick by brick and found out that the problem was I
was sending an entire foreach loop to the $parent and apparently that is
a no-no.

I am doing things in a way now that I think will work. Not terribly
sure yet, a lot more testing still needed, but I'll see.

>When you say "put that in a proc" what do you mean. Do you mean
>literallly wrapping "proc something {args} { ... } around the code
>above verbatim?

Yes.

>If so, then go back and look at what I posted and note that if you
>wrapped "the above snippet" in a proc, that you overlooked or ignored
>the line above thread::send where the "thread" variable was defined.
>It holds the thread id of the created child thread (and you need a
>thread's id value to be able to "send" anything to it. You also
>probably overlooked the other line where a thread send was used to set
>a variable 'parent' in the child thread telling it the thread id value
>of its parent.

Again, I didn't understand what you meant. Or at least I assumed you
meant I should make the thread ID variable global. I did that and it
works. Thanks.

I have a new problem now.

I need to know when the thread is done. So I adds a variable to the end
of the thread {script} then use 'vwait variable' to know it's done.

It works, except that, well...

I have a big listing of files and directories. I am looking at them.
I open a directory. The application automatically detects that one of
its subdirectories is large and needs to be cached and stored in the
"freezer." Which is done pronto. And when it's done, my "selection"
(whatever line was selected before the caching started) is still
"selected" like nothing happened. Good.

As soon as I use vwait anywhere in the caching/freezing code, that
selection is no longer kept. There is no selection anymore. So I press
an arrow key and the selection jumps to the bottom of the text widget.
Which is really weird because 1. the widget with the selection is not
involved in the caching procedure and 2. there is nothing I can imagine
that would mess with the selection, and adding vwait to the code is
really all it takes for the phenomenon to happen. I observed that
behavior in another one of the prototypes I built to investigate the
threads.

Do you have any ideas why that happens? I really need to know when the
thread is finished so I can "pack" the finished tab in a neat, properly
labeled Tupperware before it's nicely tucked into the freezer.

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> On Thu, 4 Jan 2024 06:35:18 -0000 (UTC), Rich wrote:
>
>>> Can you give me one or two examples of how I could possibly be blocking
>>> the event loop? I honestly don't know what to look for.
>>
>>Simple, anytime any of your Tcl code is running, the event loop is
>>blocked.
>
> I had no clue what you meant by that but I spent a long time rebuilding
> a lot of the app brick by brick and found out that the problem was I
> was sending an entire foreach loop to the $parent and apparently that is
> a no-no.

There's nothing wrong with sending a foreach loop, unless...

But a foreach loop is "Tcl code" -- which means that for however long
it takes for the foreach to run, the event loop is not running. I.e.,
"anytime any of your Tcl code is running (i.e., the foreach loop), the
event loop is blocked".

> I am doing things in a way now that I think will work. Not terribly
> sure yet, a lot more testing still needed, but I'll see.

If you go back and look at the example I posted, you'll see that the
child thread sends back a script of three Tcl statements. A "set"
statement (which includes a button creation), a "window create"
statement on a "text widget" and an "insert end" statement on a text
widget. Each "send" does "just one thing" (creates one button, inserts
that one button, makes sure the "end" is visible).

That is why the UI remains responsive. During the brief intervals
where those three statements are running, the UI is frozen. But those
brief intervals are tiny compared to the amount of idle time in between
send actions. All that idle time is when the event loop can run.

>>When you say "put that in a proc" what do you mean. Do you mean
>>literallly wrapping "proc something {args} { ... } around the code
>>above verbatim?
>
> Yes.
>
>
>>If so, then go back and look at what I posted and note that if you
>>wrapped "the above snippet" in a proc, that you overlooked or ignored
>>the line above thread::send where the "thread" variable was defined.
>>It holds the thread id of the created child thread (and you need a
>>thread's id value to be able to "send" anything to it. You also
>>probably overlooked the other line where a thread send was used to set
>>a variable 'parent' in the child thread telling it the thread id value
>>of its parent.
>
> Again, I didn't understand what you meant.

The code makes use of a variable $thread. If you just wrap it in a
proc, and don't create a proc local $thread variable, you'll get
"variable not found" errors.

> Or at least I assumed you meant I should make the thread ID variable
> global. I did that and it works. Thanks.

Or you pass $thread in as a proc argument:

proc run-background {thread} {
...
}

# later

run-background $thread

> I have a new problem now.
>
> I need to know when the thread is done. So I adds a variable to the end
> of the thread {script} then use 'vwait variable' to know it's done.

To the end of which "the thread" (you have two threads, the main
default one and the background one).

> It works, except that, well...
>
> I have a big listing of files and directories. I am looking at them.
> I open a directory. The application automatically detects that one of
> its subdirectories is large and needs to be cached and stored in the
> "freezer." Which is done pronto. And when it's done, my "selection"
> (whatever line was selected before the caching started) is still
> "selected" like nothing happened. Good.
>
> As soon as I use vwait anywhere in the caching/freezing code, that
> selection is no longer kept. There is no selection anymore. So I press
> an arrow key and the selection jumps to the bottom of the text widget.
> Which is really weird because 1. the widget with the selection is not
> involved in the caching procedure and 2. there is nothing I can imagine
> that would mess with the selection, and adding vwait to the code is
> really all it takes for the phenomenon to happen. I observed that
> behavior in another one of the prototypes I built to investigate the
> threads.
>
> Do you have any ideas why that happens? I really need to know when the
> thread is finished so I can "pack" the finished tab in a neat, properly
> labeled Tupperware before it's nicely tucked into the freezer.

One idea: vwait calls do not nest. You can get weird effects when you
try using a second vwait while inside another vwait (and very much so
if you are vwaiting on the same variable both times). I suspect this
is the cause, given that you say everything works correctly until you add
the vwait. Read https://wiki.tcl-lang.org/page/vwait for some of the
details.

If you really need to do something once the background work is
complete, then create a proc that does the "something" and have the
background thread call that proc (via thread::send) when it is done to
cause that additional work to happen, after the "loading" work is
finished.

On Fri, 5 Jan 2024 16:33:23 -0000 (UTC), Rich wrote:

>There's nothing wrong with sending a foreach loop, unless...

That was the problem, I was sending the whole loop to the thread,
"foreach" included, everything.

I realized my mistake and learned to do it correctly. It seems to be
working well now.

>If you really need to do something once the background work is
>complete, then create a proc that does the "something" and have the
>background thread call that proc (via thread::send) when it is done to
>cause that additional work to happen, after the "loading" work is
>finished.

I did it another way. I worked out some clever way to create a unique
ID for the thread...

[string map "/ {}" $path]
looking both ways before crossing to avoid collisions

and use that as the vwait variable at the end of the thread and it's
working.

I believe that obstacle is over. I don't see myself using threads for
anything else ahead except updating a database, but that won't be nearly
as complicated because it won't involve live widgets with images. Just
fire and forget.

Whew! I was stuck on that one for days. I gotta say, I hated every
minute of it, mostly because working out variable scope within threads
is a real nightmare. One big escape-o-rama nightmare that now works like
this then it works like that then maybe it won't work either way.
Well, when you're going through hell, keep walking. I'm glad it's over.

Now I'm working on "thawing" frozen tabs. That is a little complicated
too because the whole design is a little entangled. I may have to
redesign some of it. But I was sort of expecting that. A little cleaning
up every couple of miles along the way is needed anyway.

You helped a ton once again. I am beholden to you.

--
Luc
>>