Hi All,
After upgrade Solaris 11.1 to 11.4 with replacement of Apache 2.2 by 2.4, the browser happity shows the directories with filesystem permission 711 owned *not* by webservd
In shell as webservd user, everything works correctly - I can only traverse the directories but not list them

Puzzle... I'm sure apache runs as webservd... Filesystem is native Solaris 11.4 ZFS

On 10/10/2023 02:51, ARZ Lab wrote:
> Hi All,
> After upgrade Solaris 11.1 to 11.4 with replacement of Apache 2.2 by 2.4, the browser happity shows the directories with filesystem permission 711 owned *not* by webservd
> In shell as webservd user, everything works correctly - I can only traverse the directories but not list them
>
> Puzzle... I'm sure apache runs as webservd... Filesystem is native Solaris 11.4 ZFS

This has me intrigued, what are you doing with the browser and how is
your web space set up that it can see such things???

Cheers,
Gary B-)

On Friday, October 13, 2023 at 2:29:10 PM UTC+2, Gary R. Schmidt wrote:
> On 10/10/2023 02:51, ARZ Lab wrote:
> > Hi All,
> > After upgrade Solaris 11.1 to 11.4 with replacement of Apache 2.2 by 2.4, the browser happity shows the directories with filesystem permission 711 owned *not* by webservd
> > In shell as webservd user, everything works correctly - I can only traverse the directories but not list them
> >
> > Puzzle... I'm sure apache runs as webservd... Filesystem is native Solaris 11.4 ZFS
> This has me intrigued, what are you doing with the browser and how is
> your web space set up that it can see such things???
>
> Cheers,
> Gary B-)

Hello Gary, I'm intrigued, too! Can't crack it despite 30-years marriage with UNIX ))
It used to work just fine with Apache 2.2 in Solaris 11.1, but changed after upgrade to 11.4 and 2.4...
I made a simple and clean demo, the prod server behaves the same way

root@inet:/# zoneadm list -iv
ID NAME STATUS PATH BRAND IP
3 inet running / solaris shared

root@inet:/# uname -a
SunOS inet 5.11 11.4.42.111.0 i86pc i386 i86pc non-global-zone

I'm running in NGZ, allright

From /var/apache2/2.4/conf/httpd.conf:

User webservd
DocumentRoot "/var/apache2/2.4/htdocs"

oot@inet:/# ps -ef | grep http
webservd 15790 15789 0 Oct 09 ? 0:00 /usr/apache2/2.4/bin/httpd -k start
webservd 15793 15789 0 Oct 09 ? 0:06 /usr/apache2/2.4/bin/httpd -k start
webservd 15809 15789 0 Oct 09 ? 0:27 /usr/apache2/2.4/bin/httpd -k start
webservd 15791 15789 0 Oct 09 ? 0:03 /usr/apache2/2.4/bin/httpd -k start
webservd 15789 1308 0 Oct 09 ? 0:20 /usr/apache2/2.4/bin/httpd -k start
webservd 15792 15789 0 Oct 09 ? 0:02 /usr/apache2/2.4/bin/httpd -k start
webservd 15719 15694 0 16:39:02 pts/10 0:00 grep http

Now watch my fingers!

root@inet:/# find /var/apache2/2.4/htdocs -ls
8200 9 drwx--x--x 4 root root 4 Oct 17 18:24 /var/apache2/2.4/htdocs
8206 9 drwx--x--x 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a ##### despite unreadable parent directories...
8207 9 drwxr-xr-x 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b ##### webservd should be able to reach this directory AND read content
8208 9 d--------- 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b/c
8209 9 drwx--x--x 2 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b/c/d
8213 5 -rw-r--r-- 1 root root 5 Oct 9 10:16 /var/apache2/2.4/htdocs/a/b/c/d/e.txt ##### but should have no way to reach here!
root@inet:/#
root@inet:/# cat /var/apache2/2.4/htdocs/a/b/c/d/e.txt
haha

Yeah, root can of course

root@inet:/#
root@inet:/# su - webservd
webservd@inet:~$
webservd@inet:~$
webservd@inet:~$ id -a
uid=80(webservd) gid=80(webservd) groups=80(webservd)
webservd@inet:~$
webservd@inet:~$
webservd@inet:~$ ls -l /var/apache2/2.4/htdocs
/var/apache2/2.4/htdocs: Permission denied
total 17
webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a
/var/apache2/2.4/htdocs/a: Permission denied
total 17
webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a/b
total 17
d--------- 3 root root 3 Oct 9 10:49 c

All as expected so far...

webservd@inet:~$
webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a/b/c/d/e.txt
/var/apache2/2.4/htdocs/a/b/c/d/e.txt: Permission denied

And this too. But then...

webservd@inet:~$
webservd@inet:~$ curl http://localhost/a/b/c/d/e.txt
haha
webservd@inet:~$

What's going on with Apache? It can read *anything*, just like root! Even though running as webservd... Or isn't?

Pleeease don't tell me I must use Linux ))

Thanks for reading this, I hope you are having fun, too ))
It's not a joke, honestly!

Hi,

very strange. Cannot reproduce it on Solaris 11.4 SRU 62, x86-64,
non-global zone. Will try on SRU 42. What I find strange at a first
glance is that you seem to have Apache's config file sitting in

/var/apache2/2.4/conf/httpd.conf

on a "simple and clean demo". AFAIK, the default place of Apache 2.4's
config on Solaris 11.4 should be

/etc/apache2/2.4/...

I don't have a "conf" subdirectory in "/var/apache/2.4" at all.

Maybe "truss" gives you some more insight how Apache is capable to
access the directory and the document within the structure?

Kind regards,
Steffen

On 19.10.23 17:08, ARZ Lab wrote:
> On Friday, October 13, 2023 at 2:29:10 PM UTC+2, Gary R. Schmidt wrote:
>> On 10/10/2023 02:51, ARZ Lab wrote:
>> > Hi All,
>> > After upgrade Solaris 11.1 to 11.4 with replacement of Apache 2.2 by 2.4, the browser happity shows the directories with filesystem permission 711 owned *not* by webservd
>> > In shell as webservd user, everything works correctly - I can only traverse the directories but not list them
>> >
>> > Puzzle... I'm sure apache runs as webservd... Filesystem is native Solaris 11.4 ZFS
>> This has me intrigued, what are you doing with the browser and how is
>> your web space set up that it can see such things???
>>
>> Cheers,
>> Gary B-)
>
> Hello Gary, I'm intrigued, too! Can't crack it despite 30-years marriage with UNIX ))
> It used to work just fine with Apache 2.2 in Solaris 11.1, but changed after upgrade to 11.4 and 2.4...
> I made a simple and clean demo, the prod server behaves the same way
>
> root@inet:/# zoneadm list -iv
> ID NAME STATUS PATH BRAND IP
> 3 inet running / solaris shared
>
> root@inet:/# uname -a
> SunOS inet 5.11 11.4.42.111.0 i86pc i386 i86pc non-global-zone
>
> I'm running in NGZ, allright
>
> From /var/apache2/2.4/conf/httpd.conf:
>
> User webservd
> DocumentRoot "/var/apache2/2.4/htdocs"
>
> oot@inet:/# ps -ef | grep http
> webservd 15790 15789 0 Oct 09 ? 0:00 /usr/apache2/2.4/bin/httpd -k start
> webservd 15793 15789 0 Oct 09 ? 0:06 /usr/apache2/2.4/bin/httpd -k start
> webservd 15809 15789 0 Oct 09 ? 0:27 /usr/apache2/2.4/bin/httpd -k start
> webservd 15791 15789 0 Oct 09 ? 0:03 /usr/apache2/2.4/bin/httpd -k start
> webservd 15789 1308 0 Oct 09 ? 0:20 /usr/apache2/2.4/bin/httpd -k start
> webservd 15792 15789 0 Oct 09 ? 0:02 /usr/apache2/2.4/bin/httpd -k start
> webservd 15719 15694 0 16:39:02 pts/10 0:00 grep http
>
>
> Now watch my fingers!
>
> root@inet:/# find /var/apache2/2.4/htdocs -ls
> 8200 9 drwx--x--x 4 root root 4 Oct 17 18:24 /var/apache2/2.4/htdocs
> 8206 9 drwx--x--x 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a ##### despite unreadable parent directories...
> 8207 9 drwxr-xr-x 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b ##### webservd should be able to reach this directory AND read content
> 8208 9 d--------- 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b/c
> 8209 9 drwx--x--x 2 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b/c/d
> 8213 5 -rw-r--r-- 1 root root 5 Oct 9 10:16 /var/apache2/2.4/htdocs/a/b/c/d/e.txt ##### but should have no way to reach here!
> root@inet:/#
> root@inet:/# cat /var/apache2/2.4/htdocs/a/b/c/d/e.txt
> haha
>
> Yeah, root can of course
>
> root@inet:/#
> root@inet:/# su - webservd
> webservd@inet:~$
> webservd@inet:~$
> webservd@inet:~$ id -a
> uid=80(webservd) gid=80(webservd) groups=80(webservd)
> webservd@inet:~$
> webservd@inet:~$
> webservd@inet:~$ ls -l /var/apache2/2.4/htdocs
> /var/apache2/2.4/htdocs: Permission denied
> total 17
> webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a
> /var/apache2/2.4/htdocs/a: Permission denied
> total 17
> webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a/b
> total 17
> d--------- 3 root root 3 Oct 9 10:49 c
>
> All as expected so far...
>
> webservd@inet:~$
> webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a/b/c/d/e.txt
> /var/apache2/2.4/htdocs/a/b/c/d/e.txt: Permission denied
>
> And this too. But then...
>
> webservd@inet:~$
> webservd@inet:~$ curl http://localhost/a/b/c/d/e.txt
> haha
> webservd@inet:~$
>
> What's going on with Apache? It can read *anything*, just like root! Even though running as webservd... Or isn't?
>
> Pleeease don't tell me I must use Linux ))
>
> Thanks for reading this, I hope you are having fun, too ))
> It's not a joke, honestly!

In article <uhetso$o7gi$1@solani.org>,
Steffen Moser <usenet@steffen-moser.de> wrote:
>very strange. Cannot reproduce it on Solaris 11.4 SRU 62, x86-64,
>non-global zone. Will try on SRU 42. What I find strange at a first
>glance is that you seem to have Apache's config file sitting in
>
> /var/apache2/2.4/conf/httpd.conf
>
>on a "simple and clean demo". AFAIK, the default place of Apache 2.4's
>config on Solaris 11.4 should be
>
> /etc/apache2/2.4/...
>
>I don't have a "conf" subdirectory in "/var/apache/2.4" at all.

The manifest for pkg://solaris/web/server/apache-24 is here:
<URL:http://pkg.oracle.com/solaris/release/manifest/0/web%2Fserver%2Fapache-24@2.4.51%2C11.4-11.4.42.0.0.111.0%3A20211203T212052Z>

Best practice is possibly to drop your customizations into a single
application.conf in /etc/apache2/2.4/conf.d which automagically gets
included

John
groenveld@acm.org

Hi again,

tried to reproduce it on Solaris 11.4.42.111.0, but cannot:

curl http://localhost/a/b/c/d/e.txt

gives me Apache's permission denied output - as expected.

The only difference is the position of the httpd.conf file. In a plain
installation of Solaris 11.4 CBE, I don't find a

/var/apache2/2.4/conf/

directory.

Kind regards,
Steffen

On 19.10.23 17:08, ARZ Lab wrote:
> On Friday, October 13, 2023 at 2:29:10 PM UTC+2, Gary R. Schmidt wrote:
>> On 10/10/2023 02:51, ARZ Lab wrote:
>> > Hi All,
>> > After upgrade Solaris 11.1 to 11.4 with replacement of Apache 2.2 by 2.4, the browser happity shows the directories with filesystem permission 711 owned *not* by webservd
>> > In shell as webservd user, everything works correctly - I can only traverse the directories but not list them
>> >
>> > Puzzle... I'm sure apache runs as webservd... Filesystem is native Solaris 11.4 ZFS
>> This has me intrigued, what are you doing with the browser and how is
>> your web space set up that it can see such things???
>>
>> Cheers,
>> Gary B-)
>
> Hello Gary, I'm intrigued, too! Can't crack it despite 30-years marriage with UNIX ))
> It used to work just fine with Apache 2.2 in Solaris 11.1, but changed after upgrade to 11.4 and 2.4...
> I made a simple and clean demo, the prod server behaves the same way
>
> root@inet:/# zoneadm list -iv
> ID NAME STATUS PATH BRAND IP
> 3 inet running / solaris shared
>
> root@inet:/# uname -a
> SunOS inet 5.11 11.4.42.111.0 i86pc i386 i86pc non-global-zone
>
> I'm running in NGZ, allright
>
> From /var/apache2/2.4/conf/httpd.conf:
>
> User webservd
> DocumentRoot "/var/apache2/2.4/htdocs"
>
> oot@inet:/# ps -ef | grep http
> webservd 15790 15789 0 Oct 09 ? 0:00 /usr/apache2/2.4/bin/httpd -k start
> webservd 15793 15789 0 Oct 09 ? 0:06 /usr/apache2/2.4/bin/httpd -k start
> webservd 15809 15789 0 Oct 09 ? 0:27 /usr/apache2/2.4/bin/httpd -k start
> webservd 15791 15789 0 Oct 09 ? 0:03 /usr/apache2/2.4/bin/httpd -k start
> webservd 15789 1308 0 Oct 09 ? 0:20 /usr/apache2/2.4/bin/httpd -k start
> webservd 15792 15789 0 Oct 09 ? 0:02 /usr/apache2/2.4/bin/httpd -k start
> webservd 15719 15694 0 16:39:02 pts/10 0:00 grep http
>
>
> Now watch my fingers!
>
> root@inet:/# find /var/apache2/2.4/htdocs -ls
> 8200 9 drwx--x--x 4 root root 4 Oct 17 18:24 /var/apache2/2.4/htdocs
> 8206 9 drwx--x--x 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a ##### despite unreadable parent directories...
> 8207 9 drwxr-xr-x 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b ##### webservd should be able to reach this directory AND read content
> 8208 9 d--------- 3 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b/c
> 8209 9 drwx--x--x 2 root root 3 Oct 9 10:49 /var/apache2/2.4/htdocs/a/b/c/d
> 8213 5 -rw-r--r-- 1 root root 5 Oct 9 10:16 /var/apache2/2.4/htdocs/a/b/c/d/e.txt ##### but should have no way to reach here!
> root@inet:/#
> root@inet:/# cat /var/apache2/2.4/htdocs/a/b/c/d/e.txt
> haha
>
> Yeah, root can of course
>
> root@inet:/#
> root@inet:/# su - webservd
> webservd@inet:~$
> webservd@inet:~$
> webservd@inet:~$ id -a
> uid=80(webservd) gid=80(webservd) groups=80(webservd)
> webservd@inet:~$
> webservd@inet:~$
> webservd@inet:~$ ls -l /var/apache2/2.4/htdocs
> /var/apache2/2.4/htdocs: Permission denied
> total 17
> webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a
> /var/apache2/2.4/htdocs/a: Permission denied
> total 17
> webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a/b
> total 17
> d--------- 3 root root 3 Oct 9 10:49 c
>
> All as expected so far...
>
> webservd@inet:~$
> webservd@inet:~$ ls -l /var/apache2/2.4/htdocs/a/b/c/d/e.txt
> /var/apache2/2.4/htdocs/a/b/c/d/e.txt: Permission denied
>
> And this too. But then...
>
> webservd@inet:~$
> webservd@inet:~$ curl http://localhost/a/b/c/d/e.txt
> haha
> webservd@inet:~$
>
> What's going on with Apache? It can read *anything*, just like root! Even though running as webservd... Or isn't?
>
> Pleeease don't tell me I must use Linux ))
>
> Thanks for reading this, I hope you are having fun, too ))
> It's not a joke, honestly!

Many thanks for the suggestions, it helped to find a veird intermediate solution, but still not the root cause.
Looks like the problem is in loopback mount between GZ and NGZ.
On this server, the data under DocumentRoot must be used by several NGZs (running SMTP, Samba, Apache) , therefore are placed on a large zpool with intentionally confusing name "junk" ))
Zones mount it as follows (extract from /etc/zones/inet.xml, for example)

<filesystem special="/junk/apache" directory="/junk/apache" type="lofs">
<fsoption name="rw"/>
<fsoption name="nodevices"/>
</filesystem>

This results in following mount (see the 2nd line, the 1st line is a default mounted /var mount for comparison)

mount | egrep 'inet.*var on|inet.*apac'
/junk/zones/inet/root/var on junk/zones/inet/rpool/ROOT/11.4.42.111.1/var read/write/setuid/nodevices/rstchown/nonbmand/exec/xattr/atime/mountpoint=/junk/zones/inet/root/var/zone=inet/nozonemod/sharezone=10/dev=3990091 on Fri Dec 1 15:05:10 2023
/junk/zones/inet/root/junk/apache on /junk/apache read/write/setuid/nodevices/rstchown/zone=inet/nozonemod/sharezone=10/dev=3990010 on Fri Dec 1 15:05:10 2023

DocumentRoot in httpd.conf can be the default /var/apache2/2.4/htdocs, and pointed to with stmlinks, or modified to /junk/apache/htdocs - this doesn't make any difference

# grep ^DocumentRoot /junk/apache/httpd.conf*
/junk/apache/httpd.conf:DocumentRoot "/junk/apache/htdocs"
/junk/apache/httpd.conf_WithLinks:DocumentRoot "/var/apache2/2.4/htdocs"

it doesn't matter, Apache still does't respect filesystem ownership/permissions in both cases, unlike bash.
However, as soon as I give up on loopback mount and put the http content under real /var, permissions/ownership are respected by Apache.

I tried to play with various mount options in loopback mount, no luck.
I can't keep htdocs on "inet" zone's own /var directory (actually also mounted, but as own dataset via ZFS, not LOFS) because I need to read/write this directory from multiple NGZs, and that's only possible with LOFS to globally mouted ZFS

If I mount the Apache home as ZFS (legacy or via set mountpoint),

zfs set mountpoint=/junk/zones/inet/root/junk/apache junk/apache
or
zfs set mountpoint=legacy && mount -F zfs junk/apache /junk/zones/inet/root/junk/apache

again, everything works like a charm, but this way I can have one mountpoint only, unlike with LOFS.
Looks like something has changed in OS/ZFS from Solaris 11.1 to 11.4, or in Apache from 2.2 to 2.4. The latter sounds more probably, as bash still respects the filesystem permission on LOFS-mounted filesystem

Many thanks for the suggestions, it helped to find a weird intermediate solution, but still not the root cause.
Looks like the problem is in loopback mount between GZ and NGZ.
On this server, the data under DocumentRoot must be used by several NGZs (running SMTP, Samba, Apache) , therefore are placed on a large zpool with intentionally confusing name "junk" ))
Zones mount it as follows (extract from /etc/zones/inet.xml, for example)

<filesystem special="/junk/apache" directory="/junk/apache" type="lofs">
<fsoption name="rw"/>
<fsoption name="nodevices"/>
</filesystem>

This results in following mount (see the 2nd line, the 1st line is a default mounted /var mount for comparison)

mount | egrep 'inet.*var on|inet.*apac'
/junk/zones/inet/root/var on junk/zones/inet/rpool/ROOT/11.4.42.111.1/var read/write/setuid/nodevices/rstchown/nonbmand/exec/xattr/atime/mountpoint=/junk/zones/inet/root/var/zone=inet/nozonemod/sharezone=10/dev=3990091 on Fri Dec 1 15:05:10 2023
/junk/zones/inet/root/junk/apache on /junk/apache read/write/setuid/nodevices/rstchown/zone=inet/nozonemod/sharezone=10/dev=3990010 on Fri Dec 1 15:05:10 2023

DocumentRoot in httpd.conf can be the default /var/apache2/2.4/htdocs, and pointed to with stmlinks, or modified to /junk/apache/htdocs - this doesn't make any difference

# grep ^DocumentRoot /junk/apache/httpd.conf*
/junk/apache/httpd.conf:DocumentRoot "/junk/apache/htdocs"
/junk/apache/httpd.conf_WithLinks:DocumentRoot "/var/apache2/2.4/htdocs"

it doesn't matter, Apache still does't respect filesystem ownership/permissions in both cases, unlike bash.
However, as soon as I give up on loopback mount and put the http content under real /var, permissions/ownership are respected by Apache.

I tried to play with various mount options in loopback mount, no luck.
I can't keep htdocs on "inet" zone's own /var directory (actually also mounted, but as own dataset via ZFS, not LOFS) because I need to read/write this directory from multiple NGZs, and that's only possible with LOFS to globally mouted ZFS

If I mount the Apache home as ZFS (legacy or via set mountpoint),

zfs set mountpoint=/junk/zones/inet/root/junk/apache junk/apache
or
zfs set mountpoint=legacy && mount -F zfs junk/apache /junk/zones/inet/root/junk/apache

again, everything works like a charm, but this way I can have one mountpoint only, unlike with LOFS.
Looks like something has changed in OS/ZFS from Solaris 11.1 to 11.4, or in Apache from 2.2 to 2.4. The latter sounds more probably, as bash still respects the filesystem permission on LOFS-mounted filesystem

I've done fresh installation of Solaris 11.3 an d 11.4.42 as follows
Interactively Install from USB text install
Create and install NGZ "inet" with following loopbackmounts:
cat /etc/zones/inet.xml | tidy -xml -q | grep filesystem
<filesystem special="/junk/export/www" directory="/export/www" </filesystem>

boot it
zlogin inet
pkg exact-install solaris-minimal-server # reduced to minimal footprint
pkg install apache-24
Created the funny dir there to test

root@inet:# find /var/apache2/2.4/htdocs/a -ls
7521 9 drwx--x--x 3 root root 3 Dec 7 14:02 /var/apache2/2.4/htdocs/a
7522 9 drwxr-xr-x 3 root root 3 Dec 7 14:02 /var/apache2/2.4/htdocs/a/b
7523 9 d--------- 3 root root 3 Dec 7 14:02 /var/apache2/2.4/htdocs/a/b/c
7524 9 drwx--x--x 2 root root 3 Dec 7 14:02 /var/apache2/2.4/htdocs/a/b/c/d
7525 5 -rw-r--r-- 1 root root 5 Dec 7 14:02 /var/apache2/2.4/htdocs/a/b/c/d/e.txt

It works as expected in both 11.3, 11.4, unless htdocs moved to the looppack-mounted filesystem:

root@inet:/var/apache2/2.4# df -h /var/apache2/2.4/htdocs/a/.
Filesystem Size Used Available Capacity Mounted on
rpool/ROOT/11.4.42.111.1/var
1.78T 90.6M 1.78T 1% /var

mkdir /export/www/htdocs
cd /var/apache2/2.4/htdocs
find . cpio -dump /export/www/.
cd ..
mv htdocs htdocs.local
ln -s /export/www/htdocs .

root@inet:/var/apache2/2.4# ls -al
total 137
drwxr-xr-x 8 root sys 9 Dec 7 14:56 .
drwxr-xr-x 3 root sys 3 Dec 7 13:58 ..
drwxr-xr-x 2 root sys 6 Dec 7 13:58 cgi-bin
drwxr-xr-x 3 root sys 22 Dec 7 13:58 error
lrwxrwxrwx 1 root root 18 Dec 7 14:17 htdocs -> /export/www/htdocs
drwxr-xr-x 3 root sys 4 Dec 7 14:02 htdocs.local
drwxr-xr-x 3 root sys 179 Dec 7 13:58 icons
drwx------ 2 webservd webservd 4 Dec 7 13:59 logs
drwxr-xr-x 2 webservd webservd 2 Dec 7 13:58 proxy

root@inet:/var/apache2/2.4# df -h /var/apache2/2.4/htdocs/a/.
Filesystem Size Used Available Capacity Mounted on
/export/www 1.78T 216K 1.78T 1% /export/www

svcadm apache24 restart

Now, it works correctly in 11.3 (apache respects files/dirs ownershop/permissions).
And in 11.4, apache just ignores all these permissions and reads anything under htdocs, even though all httpd processes run as webservd user.