I want to split a hex string into bytes and compute their xor, as shown below:

werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
ea

I wonder if there is a more concise and simple method for this purpose.

Regards,
Zhao

On 2023-09-03 at 08:26 hongy...@gmail.com wrote:
> I want to split a hex string into bytes and compute their xor, as shown below:
>
> werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
> werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
> ea
>
> I wonder if there is a more concise and simple method for this purpose.

E.g.:

DATA=23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101
XOR=0
while read -N2 BYTE; do ((XOR^=0x$BYTE)) ; done < <(echo $DATA)
printf '%02X\n' $XOR

In article <8404323f-40c3-473b-a5fd-a37e7af1cb3bn@googlegroups.com>,
hongy...@gmail.com <hongyi.zhao@gmail.com> wrote:
>I want to split a hex string into bytes and compute their xor, as shown below:
>
>werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<<
>23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e
>'s/[ ]*$//' )
>werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}";
>do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
>ea
>
>I wonder if there is a more concise and simple method for this purpose.

$ gawk '{ x = 0; for (i=1; i<=NF; i++) x = xor(x,strtonum("0x" $i)); print x }' FPAT=.. <<< \
23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101

--
People sleep peaceably in their beds at night only because rough
men stand ready to do violence on their behalf.

George Orwell

"hongy...@gmail.com" <hongyi.zhao@gmail.com> writes:

> I want to split a hex string into bytes and compute their xor, as shown below:
>
> werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
> werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
> ea
>
> I wonder if there is a more concise and simple method for this
> purpose.

You could construct the expression directly. With only two bytes for
clarity:

printf '%x\n' $(($(sed 's/\(..\)/0x\1^/g' <<<2324)0))

The 0 at the end is needed because the sed command adds a trailing ^
operator.

--
Ben.

On Sunday, September 3, 2023 at 10:09:03 PM UTC+8, Ben Bacarisse wrote:
> "hongy...@gmail.com" <hongy...@gmail.com> writes:
>
> > I want to split a hex string into bytes and compute their xor, as shown below:
> >
> > werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
> > werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
> > ea
> >
> > I wonder if there is a more concise and simple method for this
> > purpose.
> You could construct the expression directly. With only two bytes for
> clarity:
>
> printf '%x\n' $(($(sed 's/\(..\)/0x\1^/g' <<<2324)0))
>
> The 0 at the end is needed because the sed command adds a trailing ^
> operator.

Good solution. I have confirmed all the approaches suggested above as follows:

werner@X10DAi:~$ gawk '{ x = 0; for (i=1; i<=NF; i++) x = xor(x,strtonum("0x" $i)); printf("%x\n", x) }' FPAT=.. <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101
ea
werner@X10DAi:~$ gawk '{ x = 0; for (i=1; i<=NF; i++) x = xor(x,strtonum("0x" $i)); printf "%x\n", x }' FPAT=.. <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101
ea
werner@X10DAi:~$ printf '%x\n' $(($(sed 's/\(..\)/0x\1^/g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101)0))
ea

> --
> Ben.

Regards,
Zhao

On 03.09.2023 08:26, hongy...@gmail.com wrote:
> I want to split a hex string into bytes and compute their xor, as shown below:
>
> werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
> werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
> ea
>
> I wonder if there is a more concise and simple method for this purpose.

You've already got a couple solutions. So just for completeness here's
a Kornshell solution without a shell loop or external commands

data=...
printf "%2x\n" $((${data//@(??)/0x\1^}0))

(functionally it's a variant of Ben's proposal). Unfortunately it seems
that bash (with extglob set) does not support the \1 in the expression?

Janis

> Regards,
> Zhao
>

On Monday, September 4, 2023 at 6:58:04 AM UTC+8, Janis Papanagnou wrote:
> On 03.09.2023 08:26, hongy...@gmail.com wrote:
> > I want to split a hex string into bytes and compute their xor, as shown below:
> >
> > werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
> > werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
> > ea
> >
> > I wonder if there is a more concise and simple method for this purpose.
> You've already got a couple solutions. So just for completeness here's
> a Kornshell solution without a shell loop or external commands
>
> data=...
> printf "%2x\n" $((${data//@(??)/0x\1^}0))
>
> (functionally it's a variant of Ben's proposal). Unfortunately it seems
> that bash (with extglob set) does not support the \1 in the expression?

What's your ksh version, and which one should I install for testing your code snippet, as listed below?

werner@X10DAi:~$ ksh
Command 'ksh' not found, but can be installed with:
sudo apt install ksh93u+m # version 1.0.0~beta.2-1, or
sudo apt install mksh # version 59c-16
> Janis
>
> > Regards,
> > Zhao
> >

On 04.09.2023 03:00, hongy...@gmail.com wrote:
> On Monday, September 4, 2023 at 6:58:04 AM UTC+8, Janis Papanagnou wrote:
>> On 03.09.2023 08:26, hongy...@gmail.com wrote:
>>> I want to split a hex string into bytes and compute their xor, as shown below:
>>>
>>> werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
>>> werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
>>> ea
>>>
>>> I wonder if there is a more concise and simple method for this purpose.
>> You've already got a couple solutions. So just for completeness here's
>> a Kornshell solution without a shell loop or external commands
>>
>> data=...
>> printf "%2x\n" $((${data//@(??)/0x\1^}0))
>>
>> (functionally it's a variant of Ben's proposal). Unfortunately it seems
>> that bash (with extglob set) does not support the \1 in the expression?
>
> What's your ksh version, and which one should I install for testing your code snippet, as listed below?

You should use any of the official or derived versions; this means
ksh93u+ (or ksh93u+m) are fine. (I wouldn't use any clone that has
only incomplete support of origional AT&T Kornshell's features.)

>
> werner@X10DAi:~$ ksh
> Command 'ksh' not found, but can be installed with:
> sudo apt install ksh93u+m # version 1.0.0~beta.2-1, or
> sudo apt install mksh # version 59c-16

Also note, BTW, that all posted solutions might not calculate the
value that you _expect_ in case that the data string has an odd
number of digits; _if_ that can happen with your data you should
take appropriate precautions, e.g. by conditionally prepending a
leading zero in case of an odd number of digits in the data.

Janis

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:

> On 03.09.2023 08:26, hongy...@gmail.com wrote:
>> I want to split a hex string into bytes and compute their xor, as shown below:
>>
>> werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
>> werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
>> ea
>>
>> I wonder if there is a more concise and simple method for this purpose.
>
> You've already got a couple solutions. So just for completeness here's
> a Kornshell solution without a shell loop or external commands
>
> data=...
> printf "%2x\n" $((${data//@(??)/0x\1^}0))

Much neater.

> (functionally it's a variant of Ben's proposal). Unfortunately it seems
> that bash (with extglob set) does not support the \1 in the
> expression?

You can use & but I think it's new:

printf '0x%X\n' $((${data//??/0x&^}0))

and it can be turned on and off via a variable so not idea in something
portable.

--
Ben.

On Monday, September 4, 2023 at 10:39:52 AM UTC+8, Ben Bacarisse wrote:
> Janis Papanagnou <janis_pap...@hotmail.com> writes:
>
> > On 03.09.2023 08:26, hongy...@gmail.com wrote:
> >> I want to split a hex string into bytes and compute their xor, as shown below:
> >>
> >> werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
> >> werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
> >> ea
> >>
> >> I wonder if there is a more concise and simple method for this purpose..
> >
> > You've already got a couple solutions. So just for completeness here's
> > a Kornshell solution without a shell loop or external commands
> >
> > data=...
> > printf "%2x\n" $((${data//@(??)/0x\1^}0))
> Much neater.
> > (functionally it's a variant of Ben's proposal). Unfortunately it seems
> > that bash (with extglob set) does not support the \1 in the
> > expression?
> You can use & but I think it's new:
>
> printf '0x%X\n' $((${data//??/0x&^}0))

I tried to understand the above code snippet with the help of chatgpt-4 as follows:

### begin quote ###
data=...
printf '0x%X\n' $((${data//??/0x&^}0))

explain the above ksh code.

The provided script line is an example of shell arithmetic expansion and parameter substitution in KornShell (ksh).

The code does the following:

1. `data=...`: This defines a variable `data`. It's supposed to have some hex value. The value isn't assigned in your example; it's merely given as ellipsis `...`.

2. `printf '0x%X\n' $((${data//??/0x&^}0))`: This is a more complex part.

- `printf '0x%X\n'`: This is a printf command that prints a number in hexadecimal format ('0x' at the beginning). The `%X` is a format specification for the hexadecimal output.

- `${data//??/0x&^}0`: This is a pattern substitution with arithmetic expansion.

- `${data//??/0x&}`: This will replace every two consecutive characters (`??`) in `data` with `0x&` (where `&` represents the matched part). So, if `data` was `1234`, after this operation it would become `0x120x34`.

- `${data//??/0x&^}0`: The `^` is bitwise XOR in ksh. It's used to XOR all the hexadecimal numbers together. The trailing `0` is necessary as the first XOR operation is between the `0` and the first replaced value.

The whole command is useful in converting multiple byte ASCII values into one big endian hexadecimal value. It opens up the byte string, calculates the XOR of all bytes, and prints out the final value in hexadecimal format.
### end quote ###

But I'm not sure if the above explanation is completely correct.
> and it can be turned on and off via a variable so not idea in something
> portable.
>
> --
> Ben.

Regards,
Zhao

"hongy...@gmail.com" <hongyi.zhao@gmail.com> writes:

> On Monday, September 4, 2023 at 10:39:52 AM UTC+8, Ben Bacarisse wrote:
>> Janis Papanagnou <janis_pap...@hotmail.com> writes:
>>
>> > On 03.09.2023 08:26, hongy...@gmail.com wrote:
>> >> I want to split a hex string into bytes and compute their xor, as shown below:
>> >>
>> >> werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
>> >> werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
>> >> ea
>> >>
>> >> I wonder if there is a more concise and simple method for this purpose.
>> >
>> > You've already got a couple solutions. So just for completeness here's
>> > a Kornshell solution without a shell loop or external commands
>> >
>> > data=...
>> > printf "%2x\n" $((${data//@(??)/0x\1^}0))
>> Much neater.
>> > (functionally it's a variant of Ben's proposal). Unfortunately it seems
>> > that bash (with extglob set) does not support the \1 in the
>> > expression?

Note that Janis is talking about bash now...

>> You can use & but I think it's new:
>>
>> printf '0x%X\n' $((${data//??/0x&^}0))
>
> I tried to understand the above code snippet with the help of
> chatgpt-4 as follows:
>
> ### begin quote ###
> data=...
> printf '0x%X\n' $((${data//??/0x&^}0))
>
> explain the above ksh code.

It's not ksh code (or not intended to be at last).

> The provided script line is an example of shell arithmetic expansion and parameter substitution in KornShell (ksh).
>
> The code does the following:
>
> 1. `data=...`: This defines a variable `data`. It's supposed to have some hex value. The value isn't assigned in your example; it's merely given as ellipsis `...`.
>
> 2. `printf '0x%X\n' $((${data//??/0x&^}0))`: This is a more complex part.
>
> - `printf '0x%X\n'`: This is a printf command that prints a number in hexadecimal format ('0x' at the beginning). The `%X` is a format specification for the hexadecimal output.
>
> - `${data//??/0x&^}0`: This is a pattern substitution with arithmetic expansion.
>
> - `${data//??/0x&}`: This will replace every two consecutive characters (`??`) in `data` with `0x&` (where `&` represents the matched part). So, if `data` was `1234`, after this operation it would become `0x120x34`.
>
> - `${data//??/0x&^}0`: The `^` is bitwise XOR in ksh. It's used to XOR all the hexadecimal numbers together. The trailing `0` is necessary as the first XOR operation is between the `0` and the first replaced value.
>
> The whole command is useful in converting multiple byte ASCII values into one big endian hexadecimal value. It opens up the byte string, calculates the XOR of all bytes, and prints out the final value in hexadecimal format.
> ### end quote ###
>
> But I'm not sure if the above explanation is completely correct.

Nor me. Why use something like ChatGPT that needs checking? Why not
check it yourself from the bash manual? Or you could ask the group to
explain it if there's some bit you don't follow.

--
Ben.

On Tuesday, September 5, 2023 at 2:47:11 AM UTC+8, Ben Bacarisse wrote:
> "hongy...@gmail.com" <hongy...@gmail.com> writes:
>
> > On Monday, September 4, 2023 at 10:39:52 AM UTC+8, Ben Bacarisse wrote:
> >> Janis Papanagnou <janis_pap...@hotmail.com> writes:
> >>
> >> > On 03.09.2023 08:26, hongy...@gmail.com wrote:
> >> >> I want to split a hex string into bytes and compute their xor, as shown below:
> >> >>
> >> >> werner@X10DAi:~$ data=$( sed 's/\(..\)/\1 /g' <<< 23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101 | sed -e 's/[ ]*$//' )
> >> >> werner@X10DAi:~$ IFS=' ' read -ra ADDR <<< "$data"; xor=0; for i in "${ADDR[@]}"; do xor=$((xor ^ 0x$i)); done; printf '%x\n' $xor
> >> >> ea
> >> >>
> >> >> I wonder if there is a more concise and simple method for this purpose.
> >> >
> >> > You've already got a couple solutions. So just for completeness here's
> >> > a Kornshell solution without a shell loop or external commands
> >> >
> >> > data=...
> >> > printf "%2x\n" $((${data//@(??)/0x\1^}0))
> >> Much neater.
> >> > (functionally it's a variant of Ben's proposal). Unfortunately it seems
> >> > that bash (with extglob set) does not support the \1 in the
> >> > expression?
> Note that Janis is talking about bash now...
> >> You can use & but I think it's new:
> >>
> >> printf '0x%X\n' $((${data//??/0x&^}0))
> >
> > I tried to understand the above code snippet with the help of
> > chatgpt-4 as follows:
> >
> > ### begin quote ###
> > data=...
> > printf '0x%X\n' $((${data//??/0x&^}0))
> >
> > explain the above ksh code.
> It's not ksh code (or not intended to be at last).
> > The provided script line is an example of shell arithmetic expansion and parameter substitution in KornShell (ksh).
> >
> > The code does the following:
> >
> > 1. `data=...`: This defines a variable `data`. It's supposed to have some hex value. The value isn't assigned in your example; it's merely given as ellipsis `...`.
> >
> > 2. `printf '0x%X\n' $((${data//??/0x&^}0))`: This is a more complex part.
> >
> > - `printf '0x%X\n'`: This is a printf command that prints a number in hexadecimal format ('0x' at the beginning). The `%X` is a format specification for the hexadecimal output.
> >
> > - `${data//??/0x&^}0`: This is a pattern substitution with arithmetic expansion.
> >
> > - `${data//??/0x&}`: This will replace every two consecutive characters (`??`) in `data` with `0x&` (where `&` represents the matched part). So, if `data` was `1234`, after this operation it would become `0x120x34`.
> >
> > - `${data//??/0x&^}0`: The `^` is bitwise XOR in ksh. It's used to XOR all the hexadecimal numbers together. The trailing `0` is necessary as the first XOR operation is between the `0` and the first replaced value.
> >
> > The whole command is useful in converting multiple byte ASCII values into one big endian hexadecimal value. It opens up the byte string, calculates the XOR of all bytes, and prints out the final value in hexadecimal format.
> > ### end quote ###
> >
> > But I'm not sure if the above explanation is completely correct.
> Nor me. Why use something like ChatGPT that needs checking? Why not
> check it yourself from the bash manual? Or you could ask the group to
> explain it if there's some bit you don't follow.

I tried to test the above code snippet as follows:

werner@X10DAi:~$ echo $SHELL
/bin/bash

werner@X10DAi:~$ data=23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101
werner@X10DAi:~$ printf '0x%X\n' $((${data//??/0x&^}0))
bash: 0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0: syntax error: operand expected (error token is "^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0")
werner@X10DAi:~$ printf '0x%X\n' $((${data//??/0x\1^}0))
0x0
> --
> Ben.

Zhao

"hongy...@gmail.com" <hongyi.zhao@gmail.com> writes:

> I tried to test the above code snippet as follows:
>
> werner@X10DAi:~$ echo $SHELL
> /bin/bash
>
> werner@X10DAi:~$ data=23230DFE4C5336413245313636504135303333333401000A130A1814300A01010101
> werner@X10DAi:~$ printf '0x%X\n' $((${data//??/0x&^}0))
> bash: 0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0: syntax error: operand expected (error token is "^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0x&^0")

I wrote:

You can use & but I think it's new:

printf '0x%X\n' $((${data//??/0x&^}0))

and it can be turned on and off via a variable so not ideal in something
portable. (I should have said "via a shell option" and I mis-typed
"ideal".)

man bash says:

If the patsub_replacement shell option is enabled using shopt,
any unquoted instances of & in string are replaced with the
matching portion of pattern.

You may have an old version of bash or you may have patsub_replacement
turned off.

> werner@X10DAi:~$ printf '0x%X\n' $((${data//??/0x\1^}0))

That's not going to work. In bash \1 is just 1 which has no special
meaning in a substitution. Every pair of digits simply gets replaced by
0x1^. The result evaluates to 0 or 1 depending on how many pairs there
are.

--
Ben.