On 2023-07-30, Christian Weisgerber <naddy@mips.inka.de> wrote:
> On 2023-07-30, Randal L. Schwartz <merlyn@stonehenge.com> wrote:
>
>> Classic Unix filesystems do *not* have a "creation date". They have a
>> "ctime" field, which updates when the file is opened for write, or the
>> metadata about the file has changed (name, number of links, and so on).
>>
>> Renaming a file traditionally has updated ctime,
>
> ... because it was implemented as a sequence of link(2) and unlink(2),
> which touched the inode's link count.
>
> I have a vague memory that the atomicity (or lack thereof) of the
> later rename(2) system call was a much discussed topic, but I don't
> remember what's the status on modern operating systems.

It's funny that something which is probably 3 lines of C somewhere
keeps generating debate decades later.

If every decision in every 3 lines came with the threat of debate
decades later, I'd be paralyzed from coding anything. :)

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Kenny McCormack wrote:

> But the real point here, which has been alluded to by various posters (but
> not stated explicitly), is that many new users (including, I believe, OP)
> assume that the "c" in "ctime" stands for "creation". In fact, it doesn't.
> It stands for "change" - specifically, an inode change, as opposed to a
> data change.
>
> And, in fact, when you rename a file (*), "ctime" *is* updated.

Some systems update it, some don't. It may well be an old SysV v. BSD
difference, as Solaris updates it but macOS does not.

--
Geoff Clare <netnews@gclare.org.uk>

On 30.07.2023 18:03, Kenny McCormack wrote:
> In article <ua5pko$2uu8g$1@dont-email.me>,
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>> This part made me curious...
>>
>> On 30.07.2023 13:49, Kenny McCormack wrote:
>>>
>>> (*) Note that, at the nuts-and-bolts level, in Unix terms, a "rename"
>>> always boils down to a link followed by an unlink.
>>
>> Inspecting the system log file during a file "rename" invoked by mv(1)
>> I can only spot a rename(2) system call (but no link(2) or unlink(2)).
>
> Obviously, somebody was going to post that.
>
> But I think even lower, at the real nuts&bolts level, rename is doing the
> equivalent of link/unlink. The point of making rename a system call was to
> make it possible to make certain guarantees of atomicity.

If atomicity is the topic then a statement "always boils down to a link
followed by an unlink" makes even less sense. But okay, YMMV.

Janis

On 2023-07-31, Geoff Clare <geoff@clare.See-My-Signature.invalid> wrote:

>> And, in fact, when you rename a file (*), "ctime" *is* updated.
>
> Some systems update it, some don't. It may well be an old SysV v. BSD
> difference, as Solaris updates it but macOS does not.

FreeBSD and OpenBSD do update it.

--
Christian "naddy" Weisgerber naddy@mips.inka.de

In article <avmkpj-up8.ln1@ID-313840.user.individual.net>,
Geoff Clare <netnews@gclare.org.uk> wrote:
>Some systems update it, some don't. It may well be an old SysV v. BSD
>difference, as Solaris updates it but macOS does not.

On Omnios (illumos) stat(1) lists an updated ctime on ZFS and tmpfs(4FS).
With ZFS stat(1) reports a birthtime.

John
groenveld@acm.org

Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
> On 30.07.2023 18:03, Kenny McCormack wrote:

>> But I think even lower, at the real nuts&bolts level, rename is doing
>> the equivalent of link/unlink. The point of making rename a system
>> call was to make it possible to make certain guarantees of atomicity.
>
> If atomicity is the topic then a statement "always boils down to a link
> followed by an unlink" makes even less sense. But okay, YMMV.

“rename is link followed by unlink, but atomic” doesn’t reflect what
happens with directories, where you need to add an additional unlink
followed by link to reflect the overall outcome (at least in traditional
Unix filesystsem).

It’s also troublesome as an explanatory model in filesystems which don’t
support hard links (i.e. which can’t have two names for one file
existing at once).

Finally it’s not a great model when talking to an audience which isn’t
familiar or comfortable with the idea of multiple names for a single
file.

With those caveats in mind, though, I agree with Kenny, and the reason
for that is that you can still see reflections of the link+unlink design
(generally surrounded by a lot of additional complexity) in some modern
kernel implementations.

--
https://www.greenend.org.uk/rjk/

Christian Weisgerber wrote:

> On 2023-07-31, Geoff Clare <geoff@clare.See-My-Signature.invalid> wrote:
>
>>> And, in fact, when you rename a file (*), "ctime" *is* updated.
>>
>> Some systems update it, some don't. It may well be an old SysV v. BSD
>> difference, as Solaris updates it but macOS does not.
>
> FreeBSD and OpenBSD do update it.

Thanks. I suppose that means either the macOS rename() internals
didn't come from FreeBSD or there has been a change in FreeBSD since
Apple used it as the basis for OS X.

--
Geoff Clare <netnews@gclare.org.uk>

In article <o5anpj-nh6.ln1@ID-313840.user.individual.net>,
Geoff Clare <netnews@gclare.org.uk> wrote:
>Christian Weisgerber wrote:
>
>> On 2023-07-31, Geoff Clare <geoff@clare.See-My-Signature.invalid> wrote:
>>
>>>> And, in fact, when you rename a file (*), "ctime" *is* updated.
>>>
>>> Some systems update it, some don't. It may well be an old SysV v. BSD
>>> difference, as Solaris updates it but macOS does not.
>>
>> FreeBSD and OpenBSD do update it.
>
>Thanks. I suppose that means either the macOS rename() internals
>didn't come from FreeBSD or there has been a change in FreeBSD since
>Apple used it as the basis for OS X.

It isn't just the OS that is the variable. The underlying filesystem
matters, too. Having used a Mac myself, I know that the underlying
filesystem (HFS) is not really a regular Unix filesystem - it has its roots
in more PC-ish OSes (i.e., original Mac OS, FAT, HPFS/NTFS, etc). For
example (and be warned that I don't have access to a Mac at the moment to
test this, so my memory may be inaccurate), but my memory is that HFS
doesn't really support hard links (they are emulated in software). Also,
it is only case retentive (like NTFS), not case sensitive (like regular
Unix/Linux filesystems).

Also, see the man page for renameat2(2), for another example of where the
underlying fs matters.

--
I am not a troll.
Rick C. Hodgin
I am not a crook.
Rick M. Nixon

In article <86edkq9p2n.fsf@red.stonehenge.com>,
Randal L. Schwartz <merlyn@stonehenge.com> wrote:
>>>>>> "Ottavio" == Ottavio Caruso <ottavio2006-usenet2012@yahoo.com> writes:
>
>Ottavio> Thanks, but it sucks. Has this been "mv" standard behaviour
>Ottavio> from the UNIX days? You would expect that renaming a file
>Ottavio> literally creates a new file, hence modifying creation date.
>
>Classic Unix filesystems do *not* have a "creation date". They have a
>"ctime" field, which updates when the file is opened for write, or the
>metadata about the file has changed (name, number of links, and so on).

I'm not sure about the "opened for write" bit. On a write(2),
sure, but not necessary an open(2); at least, this is not listed
in the man page as a call that updates `st_ctime`. A quick spin
through 4.4BSD's ufs leads me to believe `ctime` is only changed
with the inode is updated, but not on open.

- Dan C.

In article <ua5udp$2vbt2$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>On 30.07.2023 14:33, Christian Weisgerber wrote:
>> On 2023-07-30, Randal L. Schwartz <merlyn@stonehenge.com> wrote:
>>
>>> Classic Unix filesystems do *not* have a "creation date". They have a
>>> "ctime" field, which updates when the file is opened for write, or the
>>> metadata about the file has changed (name, number of links, and so on).
>>>
>>> Renaming a file traditionally has updated ctime,
>>
>> ... because it was implemented as a sequence of link(2) and unlink(2),
>> which touched the inode's link count.
>>
>> I have a vague memory that the atomicity (or lack thereof) of the
>> later rename(2) system call was a much discussed topic, but I don't
>> remember what's the status on modern operating systems.
>
>I think the "later" and the "modern" should be put in a quantified
>age context. My Stevens' APUE from 1993 (that's three decades ago)
>already describes rename(2) (and also mentions POSIX.1 changes).
>An older Unix book (based on UNIX Version 7) from 1984 didn't have
>it listed.

Research Unix up to the 7th Edition implemented `mv` as a pair
of system calls: `link` and then `unlink`. The introduction of
the 4.2BSD "fast filesystem" brought the `rename` system call
(really, this seems to have first showed up in 4.1c) which was
atomic. Curiously, post-7th Ed Unix, which reimported the 4BSD
kernel, seems to have eschewed the `rename(2)` system call.

- Dan C.

In article <87wmyhwe25.fsf@bsb.me.uk>,
Ben Bacarisse <ben.usenet@bsb.me.uk> wrote:
>gazelle@shell.xmission.com (Kenny McCormack) writes:
>
>> But the real point here, which has been alluded to by various posters (but
>> not stated explicitly), is that many new users (including, I believe, OP)
>> assume that the "c" in "ctime" stands for "creation". In fact, it doesn't.
>> It stands for "change" - specifically, an inode change, as opposed to a
>> data change.
>
>That's a new interpretation. In the original paper on Unix, the i-node
>is described as having a creation time. The field names, st_ctime,
>st_mtime and st_atime were chosen with these meanings. When (very soon
>after) a new kind of time had to be recorded, st_ctime was re-purposed.

It is true that the 1978 paper says that, as does the comment on
the `st_ctime` field in `sys/ino.h` in 7th Edition; `st_ctime`
was not present before that.

However, that is not the original Unix paper; the earlier 1974
paper published in CACM does not mention a ctime at all; instead
it just mentions "5. Time of last modification." Note that
mention of accessed time is also absent here.
https://dsf.berkeley.edu/cs262/unix.pdf

Note that with respect to 7th Edition, both the comment and the
paperappear to be wrong. Looking at the kernel sources,
`di_ctime` is updated by `iupdat` in `usr/sys/iget.c` whenever
the `ICHG` flag is set on the inode; that flag is set in a
number of places (write, chmod, chown, link/unlink, etc) in
addition to the original file creation.

>So what the "c" in ctime stands for has changed. It's no wonder people
>are confused because atime, mtime and ctime are so obvious. Had it
>originally been used for its later purpose, I doubt it would have been
>called ctime.

It would seem that the `ctime` field in the inode has always had
a meaning very close to what we assume it to be in a modern
context; if it was ever _just_ the creation time, then that
would likely have been in an unreleased version of Unix and only
for a very short time indeed.

- Dan C.

Geoff Clare <geoff@clare.See-My-Signature.invalid> writes:
> Christian Weisgerber wrote:
>> On 2023-07-31, Geoff Clare <geoff@clare.See-My-Signature.invalid> wrote:
>>>> And, in fact, when you rename a file (*), "ctime" *is* updated.
>>>
>>> Some systems update it, some don't. It may well be an old SysV v. BSD
>>> difference, as Solaris updates it but macOS does not.
>>
>> FreeBSD and OpenBSD do update it.
>
> Thanks. I suppose that means either the macOS rename() internals
> didn't come from FreeBSD or there has been a change in FreeBSD since
> Apple used it as the basis for OS X.

My understanding is that MacOS was based on an older BSD (and other
sources), not on FreeBSD. FreeBSD and MacOS are siblings or cousins,
not parent and child.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Will write code for food.
void Void(void) { Void(); } /* The recursive call of the void */

On 01.08.2023 09:40, Richard Kettlewell wrote:
> Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
>> On 30.07.2023 18:03, Kenny McCormack wrote:
>
>>> But I think even lower, at the real nuts&bolts level, rename is doing
>>> the equivalent of link/unlink. The point of making rename a system
>>> call was to make it possible to make certain guarantees of atomicity.
>>
>> If atomicity is the topic then a statement "always boils down to a link
>> followed by an unlink" makes even less sense. But okay, YMMV.
>
> “rename is link followed by unlink, but atomic” doesn’t reflect what
> happens with directories, where you need to add an additional unlink
> followed by link to reflect the overall outcome (at least in traditional
> Unix filesystsem).
>
> It’s also troublesome as an explanatory model in filesystems which don’t
> support hard links (i.e. which can’t have two names for one file
> existing at once).

Same issue with cross-device links.

Generally I agree with you concerning directories and the general case
of hard links (whether non-supporting FS or cross-device applications).

Or similar a 'mv' when considering cross-device arguments (emulated by
cp/rm).

>
> Finally it’s not a great model when talking to an audience which isn’t
> familiar or comfortable with the idea of multiple names for a single
> file.

Speaking about "models", I wouldn't consider explaining rename(2) by
link(2) and unlink(2) (or explaining mv(1) by cp(1) and rm(1), just to
make my point clear) a sensible approach if it doesn't reflect reality
(which is optimized for the simple standard cases). If an "audience"
is operating/working on _that_ OS level it should IMO know the details
and differences of the supported actual cases. A model that gives the
impression that an [atomic] system function is effectively built upon
more than one system function - thus conceptually/model-wise becoming
non-atomic - I'd not consider a preferable description. Introducing
functions that aren't actually used to explain a function I also don't
consider sensible. (The formal part of the POSIX specs, for example,
doesn't mention them; they are unnecessary for understanding.) But as
I said upthread, mileages (obviously) vary.

>
> With those caveats in mind, though, I agree with Kenny, and the reason
> for that is that you can still see reflections of the link+unlink design
> (generally surrounded by a lot of additional complexity) in some modern
> kernel implementations.

Maybe. I currently have no systems other than Linux available. Though
the POSIX description should anyway suffice.

Janis

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
> Geoff Clare <geoff@clare.See-My-Signature.invalid> writes:
>> Thanks. I suppose that means either the macOS rename() internals
>> didn't come from FreeBSD or there has been a change in FreeBSD since
>> Apple used it as the basis for OS X.
>
> My understanding is that MacOS was based on an older BSD (and other
> sources), not on FreeBSD. FreeBSD and MacOS are siblings or cousins,
> not parent and child.

At some point between NeXTStep and current macOS they upgraded from 4.x
BSD to FreeBSD.

https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/KernelProgramming/BSD/BSD.html

But this isn’t probably not directly relevant to the provenance of
rename(); the nuts and bolts are filesystem-dependent and HFS/HFS+ are
Apple code.

--
https://www.greenend.org.uk/rjk/

On Tue, 01 Aug 2023 13:40:35 -0700, Keith Thompson wrote:

> Geoff Clare <geoff@clare.See-My-Signature.invalid> writes:
>> Christian Weisgerber wrote:
>>> On 2023-07-31, Geoff Clare <geoff@clare.See-My-Signature.invalid> wrote:
>>>>> And, in fact, when you rename a file (*), "ctime" *is* updated.
>>>>
>>>> Some systems update it, some don't. It may well be an old SysV v. BSD
>>>> difference, as Solaris updates it but macOS does not.
>>>
>>> FreeBSD and OpenBSD do update it.
>>
>> Thanks. I suppose that means either the macOS rename() internals
>> didn't come from FreeBSD or there has been a change in FreeBSD since
>> Apple used it as the basis for OS X.
>
> My understanding is that MacOS was based on an older BSD (and other
> sources), not on FreeBSD. FreeBSD and MacOS are siblings or cousins,
> not parent and child.

macOS version history in WIKI has a section on Development outside Apple
that is consistent with what I have read and remember over the years.

en.wikipedia.org/wiki/MacOS_version_history

Cheers!

jim b.

--
UNIX is not user-unfriendly, it merely
expects users to be computer friendly.

In article <uabr7k$3ospu$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
>[snip]
>Speaking about "models", I wouldn't consider explaining rename(2) by
>link(2) and unlink(2) (or explaining mv(1) by cp(1) and rm(1), just to
>make my point clear) a sensible approach if it doesn't reflect reality
>(which is optimized for the simple standard cases). If an "audience"
>is operating/working on _that_ OS level it should IMO know the details
>and differences of the supported actual cases. A model that gives the
>impression that an [atomic] system function is effectively built upon
>more than one system function - thus conceptually/model-wise becoming
>non-atomic - I'd not consider a preferable description. Introducing
>functions that aren't actually used to explain a function I also don't
>consider sensible. (The formal part of the POSIX specs, for example,
>doesn't mention them; they are unnecessary for understanding.) But as
>I said upthread, mileages (obviously) vary.

It is illustrative here to look at the history of the call, and
when it was added in BSD (4.1c is when it first appeared). From
sys/sys/ufs_syscalls.c, we can see the comment on `rename`:

/*
* Rename system call.
* rename("foo", "bar");
* is essentially
* unlink("bar");
* link("foo", "bar");
* unlink("foo");
* but ``atomically''. Can't do full commit without saving state in the
* inode on disk which isn't feasible at this time. Best we can do is
* always guarantee the target exists.
*
* Basic algorithm is:
*
* 1) Bump link count on source while we're linking it to the
* target. This also insure the inode won't be deleted out
* from underneath us while we work.
* 2) Link source to destination. If destination already exists,
* delete it first.
* 3) Unlink source reference to inode if still around.
* 4) If a directory was moved and the parent of the destination
* is different from the source, patch the ".." entry in the
* directory.
*
* Source and destination must either both be directories, or both
* not be directories. If target is a directory, it must be empty.
*/
(https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/a/sys/sys/ufs_syscalls.c)

In order to understand the semantics here, it is important to
contextualize what "atomic" means in the context of filesystems:
this means that an operation with side effects cannot allow an
observer to see intermediate states, one either sees the new
state or the old state; whether the transition from one to the
other requires many steps is immaterial: the point is that this
invariant is maintained _with respect to the external
observer_.

A way to unify this with a mental model may be to describe it as
being multiple operations that happen in a critical section.

Here, the comment gives us a hint about the limitations: "Best
we can do is always guarantee the target exists." Note that
this implies that both the target _and_ source may exist if,
say, the system crashes in the middle of the syscall.

- Dan C.