The Tcl sqlite3 package (from tclsqlite3.c) takes everything after the $ including the enclosing braces, and passes it to Tcl_GetVar2Ex as the variable name.

The Tcl interpreter uses Tcl_ParseVarName which checks if the character after the $ is a "{", and if it is takes everything between the matching braces as the variable name.

Dave B

clt.to.davebr@dfgh.net writes:

> The Tcl sqlite3 package (from tclsqlite3.c) takes everything after the $
> including the enclosing braces, and passes it to Tcl_GetVar2Ex as the
> variable name.
>
> The Tcl interpreter uses Tcl_ParseVarName which checks if the character
> after the $ is a "{", and if it is takes everything between the matching
> braces as the variable name.

This is concerning the thread I started?

I would find it logical if it would work this way. If at one place
$variable is the same as ${variable}, I would expect it everywhere to
be the case.
But sadly it is not.
Tho show it I use this code:
puts "Before first select"
db eval { SELECT 37 } {}
set testVal 37
puts "Before second select"
db eval { SELECT $testVal } {}
puts "Before third select"
db eval { SELECT ${testVal} } {}
puts "After selects"

When I run it I get:
Before first select
Before second select
Before third select
unrecognized token: "$"
while executing
"db eval { SELECT ${testVal} } {}"
(file "/home/cecil/bin/ytWeek.tcl" line 52)

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

On 10/26/2023 9:31 AM, Cecil Westerhof wrote:
> clt.to.davebr@dfgh.net writes:
>
>> The Tcl sqlite3 package (from tclsqlite3.c) takes everything after the $
>> including the enclosing braces, and passes it to Tcl_GetVar2Ex as the
>> variable name.
>>
>> The Tcl interpreter uses Tcl_ParseVarName which checks if the character
>> after the $ is a "{", and if it is takes everything between the matching
>> braces as the variable name.
>
> This is concerning the thread I started?
>
> I would find it logical if it would work this way. If at one place
> $variable is the same as ${variable}, I would expect it everywhere to
> be the case.
> But sadly it is not.
> Tho show it I use this code:
> puts "Before first select"
> db eval { SELECT 37 } {}
> set testVal 37
> puts "Before second select"
> db eval { SELECT $testVal } {}
> puts "Before third select"
> db eval { SELECT ${testVal} } {}
> puts "After selects"
>
> When I run it I get:
> Before first select
> Before second select
> Before third select
> unrecognized token: "$"
> while executing
> "db eval { SELECT ${testVal} } {}"
> (file "/home/cecil/bin/ytWeek.tcl" line 52)
>

If you enclose your sqlite3 statement inside of braces, then tcl doesn't parse anything inside the braces, except to keep a level count of braces so it can find the final closing brace (with \{ or \} not counted).

That's it, everything else is passed on to sqlite3 as is. See rule 6 of the 12. So, it's up to sqlite3 to deal with the $ however it chooses to. And it does not (appear) to support the ${var} format that tcl does. So, it's not tcl's fault, since sqlite3 has it's own language. It supports $, :, and @ however.

Since sqlite3 statements are generally encoded as tcl words that begin and end with {}'s, tcl $ substitution does not occur. To get ${var} handled the sqlite3 statement would need to be in double quotes (assuming it has some spaces) and then tcl would be doing the $ substitution before handing it off to sqlite3.

note for the curious:

Actually, it's Tcl_ParseVarName that handles $var, ${var}, $() or $ (as just text). The code has a comment block which explains it. It's kinda complex, since var can have namespace characters using :: and empty array names have explicit code to handle those cases.

However, the manual entry for Tcl_ParseVarName (still) doesn't mention all the possibilities, and only gives 2 examples, one of a scalar variable on one as an array. I don't know if that's a manual bug, or just a desire to not overburden the reader. The ${var} form is mentioned (only) in the tcl rule 8 of the 12 afaik.

The tcl function Tcl_GetVar2Ex is called with a pointer to the variable or array name already separated from any $.

et99 schrieb am Donnerstag, 26. Oktober 2023 um 19:30:18 UTC+2:
> On 10/26/2023 9:31 AM, Cecil Westerhof wrote:
> > clt.to:
> >
> >> The Tcl sqlite3 package (from tclsqlite3.c) takes everything after the $
> >> including the enclosing braces, and passes it to Tcl_GetVar2Ex as the
> >> variable name.
> >>
> >> The Tcl interpreter uses Tcl_ParseVarName which checks if the character
> >> after the $ is a "{", and if it is takes everything between the matching
> >> braces as the variable name.
> >
> > This is concerning the thread I started?
> >
> > I would find it logical if it would work this way. If at one place
> > $variable is the same as ${variable}, I would expect it everywhere to
> > be the case.
> > But sadly it is not.
> > Tho show it I use this code:
> > puts "Before first select"
> > db eval { SELECT 37 } {}
> > set testVal 37
> > puts "Before second select"
> > db eval { SELECT $testVal } {}
> > puts "Before third select"
> > db eval { SELECT ${testVal} } {}
> > puts "After selects"
> >
> > When I run it I get:
> > Before first select
> > Before second select
> > Before third select
> > unrecognized token: "$"
> > while executing
> > "db eval { SELECT ${testVal} } {}"
> > (file "/home/cecil/bin/ytWeek.tcl" line 52)
> >
> If you enclose your sqlite3 statement inside of braces, then tcl doesn't parse anything inside the braces, except to keep a level count of braces so it can find the final closing brace (with \{ or \} not counted).
>
> That's it, everything else is passed on to sqlite3 as is. See rule 6 of the 12. So, it's up to sqlite3 to deal with the $ however it chooses to. And it does not (appear) to support the ${var} format that tcl does. So, it's not tcl's fault, since sqlite3 has it's own language. It supports $, :, and @ however.
>

from
https://www.sqlite.org/draft/tokenreq.html

Variables are used as placeholders in SQL statements for constant values that are to be bound at start-time.
H40310: SQLite shall recognize as a VARIABLE token the a question-mark (u003f) followed by zero or more NUMERIC characters.
A "parameter name" is defined to be a sequence of one or more characters that consists of ALPHANUMERIC characters and/or dollar-signs (u0025) intermixed with pairs of colons (u003a) and optionally followed by any sequence of non-zero, non-WHITESPACE characters enclosed in parentheses (u0028 and u0029).

On 10/26/2023 10:39 AM, greg wrote:
> et99 schrieb am Donnerstag, 26. Oktober 2023 um 19:30:18 UTC+2:
>> On 10/26/2023 9:31 AM, Cecil Westerhof wrote:
>>> clt.to:
>>>
>>>> The Tcl sqlite3 package (from tclsqlite3.c) takes everything after the $
>>>> including the enclosing braces, and passes it to Tcl_GetVar2Ex as the
>>>> variable name.
>>>>
>>>> The Tcl interpreter uses Tcl_ParseVarName which checks if the character
>>>> after the $ is a "{", and if it is takes everything between the matching
>>>> braces as the variable name.
>>>
>>> This is concerning the thread I started?
>>>
>>> I would find it logical if it would work this way. If at one place
>>> $variable is the same as ${variable}, I would expect it everywhere to
>>> be the case.
>>> But sadly it is not.
>>> Tho show it I use this code:
>>> puts "Before first select"
>>> db eval { SELECT 37 } {}
>>> set testVal 37
>>> puts "Before second select"
>>> db eval { SELECT $testVal } {}
>>> puts "Before third select"
>>> db eval { SELECT ${testVal} } {}
>>> puts "After selects"
>>>
>>> When I run it I get:
>>> Before first select
>>> Before second select
>>> Before third select
>>> unrecognized token: "$"
>>> while executing
>>> "db eval { SELECT ${testVal} } {}"
>>> (file "/home/cecil/bin/ytWeek.tcl" line 52)
>>>
>> If you enclose your sqlite3 statement inside of braces, then tcl doesn't parse anything inside the braces, except to keep a level count of braces so it can find the final closing brace (with \{ or \} not counted).
>>
>> That's it, everything else is passed on to sqlite3 as is. See rule 6 of the 12. So, it's up to sqlite3 to deal with the $ however it chooses to. And it does not (appear) to support the ${var} format that tcl does. So, it's not tcl's fault, since sqlite3 has it's own language. It supports $, :, and @ however.
>>
>
> from
> https://www.sqlite.org/draft/tokenreq.html
>
> Variables are used as placeholders in SQL statements for constant values that are to be bound at start-time.
> H40310: SQLite shall recognize as a VARIABLE token the a question-mark (u003f) followed by zero or more NUMERIC characters.
> A "parameter name" is defined to be a sequence of one or more characters that consists of ALPHANUMERIC characters and/or dollar-signs (u0025) intermixed with pairs of colons (u003a) and optionally followed by any sequence of non-zero, non-WHITESPACE characters enclosed in parentheses (u0028 and u0029).
>
>

Note that sqlite3 is not just a tcl extension. When one reads the extraordinary testing procedures used, it is clear that the developers are looking at use far beyond just tcl.

So sqlite3 doesn't support every language's idiosyncratic syntax such as tcl's ${var} form.

And this form was most likely intended (only) to extend the $ operator to variable name construction not normally found in most languages, such as punctuation characters and the invisible name ${}. It was to have $var as a shortcut for [set var].

And I am quite dismayed that there is actually special purpose code to handle invisible array variable names. And partly because of this, the jimtcl extension of $(expression) for [expr {expression}] will most likely be rejected for inclusion in tcl 9.0.

And $(expression) solves the long standing problem of bracing expressions, not to mention its more readable and writable form.

Cecil Westerhof <Cecil@decebal.nl> writes:
> clt.to.davebr@dfgh.net writes:
>
>> The Tcl sqlite3 package (from tclsqlite3.c) takes everything after the $
>> including the enclosing braces, and passes it to Tcl_GetVar2Ex as the
>> variable name.
>>
>> The Tcl interpreter uses Tcl_ParseVarName which checks if the character
>> after the $ is a "{", and if it is takes everything between the matching
>> braces as the variable name.
>
> This is concerning the thread I started?

Yes. It explains why sqlite3 does not work in this respect as you
expect.

> I would find it logical if it would work this way. If at one place
> $variable is the same as ${variable}, I would expect it everywhere to
> be the case.
> But sadly it is not.

But it is.

In code evaluated by the Tcl interpreter (citing you) "$variable is the
same as ${variable}"

But the argument given to [db eval arg] isn't evaluated by the Tcl
interpreter. It is processed by the database command created by the
extension command sqlite3.

A database command created by sqlite3 expects in this case a SQL expression in
sqlite3 SQL dialect. Of course this outstandig excellent piece of
software provides a way to "inject" string literals and in a secure way.
But this commands expect the syntax $variable (or :variable) and see
${variable} as invalid syntax. That's OK. Every command is free to
interpret its arguments.

The "injection attack" - think not only about bad guys but also
about bad luck, it is a data driven bug - is a problem for every
embedded language. As Dave wrote, the Tcl API provides functions to help
extension writers so solve this (Tcl_ParseVarName, Tcl_ParseVar). I give
an example with tDOM below (the embedded query language here is XPath)
were in the embedded language $variable is the same as ${variable}.

That all said this remains to be a quirk more to your habit to prefer
${variable} over $variable.

rolf

package require tdom

set xml {
<doc>
<elem key="foo">foovalue</elem>
<elem key="bar">barvalue</elem>
</doc>
}

dom parse $xml doc
set thiskey bar
puts [$doc selectNodes {string(/doc/elem[@key=$thiskey])}]
puts [$doc selectNodes {string(/doc/elem[@key=${thiskey}])}]

Rolf Ade <rolf@pointsman.de> writes:

> Cecil Westerhof <Cecil@decebal.nl> writes:
>> clt.to.davebr@dfgh.net writes:
>>
>>> The Tcl sqlite3 package (from tclsqlite3.c) takes everything after the $
>>> including the enclosing braces, and passes it to Tcl_GetVar2Ex as the
>>> variable name.
>>>
>>> The Tcl interpreter uses Tcl_ParseVarName which checks if the character
>>> after the $ is a "{", and if it is takes everything between the matching
>>> braces as the variable name.

English is not my first language. But the above gave me the idea that
also for SQLite $var and ${var} should work.

But I am going to use :var, so it is no biggy.

--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof