There doesn't seem to be an easy answer to my query about not having
Robocopy write files when the source and destination have identical
contents but different timestamps. The main reason I cared was not
wanting to wear out flash memory prematurely.

That leads me to ask: is there any way to tell when a flash drive is
getting close to needing replacement, before it actually starts to
accumulate data errors?

--
Stan Brown, Tehachapi, California, USA https://BrownMath.com/
Shikata ga nai...

Stan Brown wrote:
> There doesn't seem to be an easy answer to my query about not having
> Robocopy write files when the source and destination have identical
> contents but different timestamps. The main reason I cared was not
> wanting to wear out flash memory prematurely.
>
> That leads me to ask: is there any way to tell when a flash drive is
> getting close to needing replacement, before it actually starts to
> accumulate data errors?
>

Short answer; No.
You get different reports from different sources, different
manufacturers, different users.
It's similar to SSDs, DVDs, HDs etc. Some have been dropped in the sea
in transit, some stored in damp warehouses, some arrived securely. You
even get some where a manufacturer operative sneezed while making it.

Your best friend is Checkdisk. See its bad sector report.

Ed

On 2023-10-25 22:55, Stan Brown wrote:
> There doesn't seem to be an easy answer to my query about not having
> Robocopy write files when the source and destination have identical
> contents but different timestamps. The main reason I cared was not
> wanting to wear out flash memory prematurely.
>
> That leads me to ask: is there any way to tell when a flash drive is
> getting close to needing replacement, before it actually starts to
> accumulate data errors?

You can check the SMART information on the disk.

<https://en.wikipedia.org/wiki/Self-Monitoring,_Analysis_and_Reporting_Technology>

I use smartmontools in Linux to see it, and it also has a Windows tool,
but I have not tried it.

https://en.wikipedia.org/wiki/Comparison_of_S.M.A.R.T._tools

https://en.wikipedia.org/wiki/Smartmontools

--
Cheers,
Carlos E.R.

On 10/25/23 05:11 PM, this is what Ed Cryer wrote:
> Stan Brown wrote:
>> There doesn't seem to be an easy answer to my query about not having
>> Robocopy write files when the source and destination have identical
>> contents but different timestamps. The main reason I cared was not
>> wanting to wear out flash memory prematurely.
>>
>> That leads me to ask: is there any way to tell when a flash drive is
>> getting close to needing replacement, before it actually starts to
>> accumulate data errors?
>>
>
> Short answer; No.
> You get different reports from different sources, different manufacturers, different users.
> It's similar to SSDs, DVDs, HDs etc. Some have been dropped in the sea in transit, some stored in damp warehouses, some
> arrived securely. You even get some where a manufacturer operative sneezed while making it.
>
> Your best friend is Checkdisk. See its bad sector report.
>
> Ed
>
http://kvipu.com/CDCheck/download.php
I've used CDCheck for years. It will text any folder and below for readability. Naturally you would just pick the
drive letter and let it rip. Unfortunately it just tests to see if files can be read, and it's last update was 2008 I
think looking at that page. But it will still work.
--
Linux Mint 21.2 Cinnamon
Al

On 10/25/2023 5:28 PM, Carlos E. R. wrote:
> On 2023-10-25 22:55, Stan Brown wrote:
>> There doesn't seem to be an easy answer to my query about not having
>> Robocopy write files when the source and destination have identical
>> contents but different timestamps. The main reason I cared was not
>> wanting to wear out flash memory prematurely.
>>
>> That leads me to ask: is there any way to tell when a flash drive is
>> getting close to needing replacement, before it actually starts to
>> accumulate data errors?
>
> You can check the SMART information on the disk.
>
> <https://en.wikipedia.org/wiki/Self-Monitoring,_Analysis_and_Reporting_Technology>
>
> I use smartmontools in Linux to see it, and it also has a Windows tool, but I have not tried it.
>
> https://en.wikipedia.org/wiki/Comparison_of_S.M.A.R.T._tools
>
> https://en.wikipedia.org/wiki/Smartmontools
>
>

I don't think a USB flash stick (UFD) has SMART. It would
only have SMART, if it was one of the sticks that consisted of
an SSD controller and a USB to SATA adapter, all inside the
stick. And the USB to SATA would need SMART passthru to work.

USB flash sticks are relatively dumb, at least there's
no sign of the sophistication of a regular drive.

SSDs and HDDs have SMART. SMART has not always existed, and
my 2GB and 4GB drives probably don't have it. The 4GB drive
writes at 5MB/sec and it still works.

The locations of things in SMART, differs on SSD and HDD, so
you have to be careful the tools you use are not too old.

SMART assumes "issues" with a storage device, are well distributed
over the surface. If a flaw in a storage, is laser focused on
a tiny portion of the surface, there can be a more-or-less
functional failure there, without SMART indicating anything is
wrong.

*******

But with respect to the OPs question, USB drives can degrade "scary fast",
so no, there is not much of a warning. I've lost a couple sticks,
and attempts to copy off one of them (as a test), the stick died completely
before I could finish. Once they drop to 1.5MB/sec I/O, you're pretty
well screwed.

Paul

On 2023-10-26 00:43, Paul wrote:
> On 10/25/2023 5:28 PM, Carlos E. R. wrote:
>> On 2023-10-25 22:55, Stan Brown wrote:
>>> There doesn't seem to be an easy answer to my query about not having
>>> Robocopy write files when the source and destination have identical
>>> contents but different timestamps. The main reason I cared was not
>>> wanting to wear out flash memory prematurely.
>>>
>>> That leads me to ask: is there any way to tell when a flash drive is
>>> getting close to needing replacement, before it actually starts to
>>> accumulate data errors?
>>
>> You can check the SMART information on the disk.
>>
>> <https://en.wikipedia.org/wiki/Self-Monitoring,_Analysis_and_Reporting_Technology>
>>
>> I use smartmontools in Linux to see it, and it also has a Windows tool, but I have not tried it.
>>
>> https://en.wikipedia.org/wiki/Comparison_of_S.M.A.R.T._tools
>>
>> https://en.wikipedia.org/wiki/Smartmontools
>>
>>
>
> I don't think a USB flash stick (UFD) has SMART.

No, I was thinking of proper media, SSD.
I assumed the OP was asking about proper media, not USB sticks.

> It would
> only have SMART, if it was one of the sticks that consisted of
> an SSD controller and a USB to SATA adapter, all inside the
> stick. And the USB to SATA would need SMART passthru to work.
>
> USB flash sticks are relatively dumb, at least there's
> no sign of the sophistication of a regular drive.
>
> SSDs and HDDs have SMART. SMART has not always existed, and
> my 2GB and 4GB drives probably don't have it. The 4GB drive
> writes at 5MB/sec and it still works.

SSDs all have it. SMART is technology from 1995..2005.

>
> The locations of things in SMART, differs on SSD and HDD, so
> you have to be careful the tools you use are not too old.
>
> SMART assumes "issues" with a storage device, are well distributed
> over the surface. If a flaw in a storage, is laser focused on
> a tiny portion of the surface, there can be a more-or-less
> functional failure there, without SMART indicating anything is
> wrong.
>
> *******
>
> But with respect to the OPs question, USB drives can degrade "scary fast",
> so no, there is not much of a warning. I've lost a couple sticks,
> and attempts to copy off one of them (as a test), the stick died completely
> before I could finish. Once they drop to 1.5MB/sec I/O, you're pretty
> well screwed.
>
> Paul

--
Cheers,
Carlos E.R.

On 25/10/2023 23:53, Carlos E. R. wrote:
> On 2023-10-26 00:43, Paul wrote:
>> It would
>> only have SMART, if it was one of the sticks that consisted of
>> an SSD controller and a USB to SATA adapter, all inside the
>> stick. And the USB to SATA would need SMART passthru to work.
>>
>> USB flash sticks are relatively dumb, at least there's
>> no sign of the sophistication of a regular drive.
>>
>> SSDs and HDDs have SMART. SMART has not always existed, and
>> my 2GB and 4GB drives probably don't have it. The 4GB drive
>> writes at 5MB/sec and it still works.
>
> SSDs all have it. SMART is technology from 1995..2005.

I was annoyed to discover that my new Seagate 4TB NVME SSD doesn't seem
to have SMART. There's a crummy Seatools program that looks and works
like it was written by an unpaid intern in his last week at Seagate. It
tells you a few things like percentage of life remaining. But SMARTCTL
basically only returns a couple of pages of information, none of which
is SMART attributes, not even ones it is unsure of the meaning of.

--
Brian Gregory (in England).

On 26/10/2023 01:24, Brian Gregory wrote:
> I was annoyed to discover that my new Seagate 4TB NVME SSD doesn't seem
> to have SMART. There's a crummy Seatools program that looks and works
> like it was written by an unpaid intern in his last week at Seagate. It
> tells you a few things like percentage of life remaining. But SMARTCTL
> basically only returns a couple of pages of information, none of which
> is SMART attributes, not even ones it is unsure of the meaning of.

Oh, I should say
SMARTCTL -H c:
does say it's PASSED its SMART overall-health self-assessment test.

--
Brian Gregory (in England).

On 2023-10-26 02:24, Brian Gregory wrote:
> On 25/10/2023 23:53, Carlos E. R. wrote:
>> On 2023-10-26 00:43, Paul wrote:
>>> It would
>>> only have SMART, if it was one of the sticks that consisted of
>>> an SSD controller and a USB to SATA adapter, all inside the
>>> stick. And the USB to SATA would need SMART passthru to work.
>>>
>>> USB flash sticks are relatively dumb, at least there's
>>> no sign of the sophistication of a regular drive.
>>>
>>> SSDs and HDDs have SMART. SMART has not always existed, and
>>> my 2GB and 4GB drives probably don't have it. The 4GB drive
>>> writes at 5MB/sec and it still works.
>>
>> SSDs all have it. SMART is technology from 1995..2005.
>
> I was annoyed to discover that my new Seagate 4TB NVME SSD doesn't seem
> to have SMART. There's a crummy Seatools program that looks and works
> like it was written by an unpaid intern in his last week at Seagate. It
> tells you a few things like percentage of life remaining. But SMARTCTL
> basically only returns a couple of pages of information, none of which
> is SMART attributes, not even ones it is unsure of the meaning of.
>

Problem with NVME is, they don't follow the SATA interface protocols,
they are not "disks". They are more like directly addressable memory
across some sort of serial? connector. They don't answer questions like
give me the smart data, someone has to find the way to ask them first.
Specially people like the guy(s) doing smartmontools, without serious
money backing him, with opensource code, reverse engineering.

I'm fortunate that the one on this laptop responds, although I don't see
a parameter saying % of life like I see on SSD I have.

On Linux, there is also the "nvme" command.

nvme smart-log /dev/nvme0n1

but little is known on it.

--
Cheers,
Carlos E.R.

On Wed, 25 Oct 2023 18:43:13 -0400, Paul wrote:
> But with respect to the OPs question, USB drives can degrade "scary fast",
> so no, there is not much of a warning. I've lost a couple sticks,
> and attempts to copy off one of them (as a test), the stick died completely
> before I could finish. Once they drop to 1.5MB/sec I/O, you're pretty
> well screwed.

Thanks. I guess I'll have to keep an eye on the
transfer speed.

I don't ind buying the occasional new USB stick; I just
don't want to copy corrupted versions of files from a
bad one to my hard drive.

--
Stan Brown, Tehachapi, California, USA
https://BrownMath.com/
Shikata ga nai...

On 10/26/2023 11:45 AM, Stan Brown wrote:
> On Wed, 25 Oct 2023 18:43:13 -0400, Paul wrote:
>> But with respect to the OPs question, USB drives can degrade "scary fast",
>> so no, there is not much of a warning. I've lost a couple sticks,
>> and attempts to copy off one of them (as a test), the stick died completely
>> before I could finish. Once they drop to 1.5MB/sec I/O, you're pretty
>> well screwed.
>
> Thanks. I guess I'll have to keep an eye on the
> transfer speed.
>
> I don't ind buying the occasional new USB stick; I just
> don't want to copy corrupted versions of files from a
> bad one to my hard drive.

I think you would be very pleased, with the behavior of SLC or MLC
sticks. They seem to last. These are usually small capacity, for the
ones that are reliably available (Mouser, Digikey, etc). While
adverts have offered 32GB devices of this type (and I was actually
able to verify that Micron had a chip like that on its site), there
is little evidence Micron actually shipped that chip. The one picture
could have been a fake (an advert which said "out of stock" from day one
and offered an inferior substitute).

TLC or QLC seem to be the norm now,
and the other ones are only "teased" in Amazon adverts.

While you can find web forum posts extolling certain devices ["Wear Leveling"],
with a lot of storage electronics, the controller could
change on a moments notice. Even items packed in plastic
packs hanging on hooks at the store, with the SAME part number,
can be from different countries, and they don't seem to work
the same. (I bought two, one had a LED, the other did not.
Speed seemed different.)

https://en.wikipedia.org/wiki/Wear_leveling

Item Static Dynamic
--------- ---------------------- -------
Endurance Longer life expectancy Shorter life expectancy
Performance Slower Faster
Design complexity More complex Less complex
Typical use SSDs,[2] industrial-grade flash drives[12] Consumer-grade flash drives

Paul

On 10/26/2023 6:27 PM, Paul wrote:
> On 10/26/2023 11:45 AM, Stan Brown wrote:
>> On Wed, 25 Oct 2023 18:43:13 -0400, Paul wrote:
>>> But with respect to the OPs question, USB drives can degrade "scary fast",
>>> so no, there is not much of a warning. I've lost a couple sticks,
>>> and attempts to copy off one of them (as a test), the stick died completely
>>> before I could finish. Once they drop to 1.5MB/sec I/O, you're pretty
>>> well screwed.
>>
>> Thanks. I guess I'll have to keep an eye on the
>> transfer speed.
>>
>> I don't ind buying the occasional new USB stick; I just
>> don't want to copy corrupted versions of files from a
>> bad one to my hard drive.
>
> I think you would be very pleased, with the behavior of SLC or MLC
> sticks. They seem to last. These are usually small capacity, for the
> ones that are reliably available (Mouser, Digikey, etc). While
> adverts have offered 32GB devices of this type (and I was actually
> able to verify that Micron had a chip like that on its site), there
> is little evidence Micron actually shipped that chip. The one picture
> could have been a fake (an advert which said "out of stock" from day one
> and offered an inferior substitute).
>
> TLC or QLC seem to be the norm now,
> and the other ones are only "teased" in Amazon adverts.
>
> While you can find web forum posts extolling certain devices ["Wear Leveling"],
> with a lot of storage electronics, the controller could
> change on a moments notice. Even items packed in plastic
> packs hanging on hooks at the store, with the SAME part number,
> can be from different countries, and they don't seem to work
> the same. (I bought two, one had a LED, the other did not.
> Speed seemed different.)
>
> https://en.wikipedia.org/wiki/Wear_leveling
>
> Item Static Dynamic
> --------- ---------------------- -------
> Endurance Longer life expectancy Shorter life expectancy
> Performance Slower Faster
> Design complexity More complex Less complex
> Typical use SSDs,[2] industrial-grade flash drives[12] Consumer-grade flash drives
>

From the year 2010.

There is one short description of what is happening near to device death.

https://www.usenix.org/legacy/event/fast10/tech/full_papers/boboila.pdf

Paul

On 10/25/23 18:25, Carlos E. R. wrote:
> nvme smart-log /dev/nvme0n1

Hi Carlos.

Now that command I do love! Thank you!

Hmmmmmm: media_errors = 0 Yippee!

-T

# nvme smart-log /dev/nvme0n1
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning : 0
temperature : 34 °C (307 K)
available_spare : 100%
available_spare_threshold : 10%
percentage_used : 2%
endurance group critical warning summary: 0
Data Units Read : 1,163,123,052 (595.52 TB)
Data Units Written : 60,234,320 (30.84 TB)
host_read_commands : 3,597,716,397
host_write_commands : 1,230,457,952
controller_busy_time : 9,407
power_cycles : 2,446
power_on_hours : 5,495
unsafe_shutdowns : 83
media_errors : 0
num_err_log_entries : 3,400
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 34 °C (307 K)
Temperature Sensor 2 : 41 °C (314 K)
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0

On 2023-10-27 07:41, T wrote:
> On 10/25/23 18:25, Carlos E. R. wrote:
>> nvme smart-log /dev/nvme0n1
>
> Hi Carlos.
>
> Now that command I do love!  Thank you!
>
> Hmmmmmm:  media_errors = 0   Yippee!
>

Welcome :-)

Now we have to find if there is a Windows version ;-)

--
Cheers,
Carlos E.R.

On 10/27/23 04:30, Carlos E. R. wrote:
> On 2023-10-27 07:41, T wrote:
>> On 10/25/23 18:25, Carlos E. R. wrote:
>>> nvme smart-log /dev/nvme0n1
>>
>> Hi Carlos.
>>
>> Now that command I do love!  Thank you!
>>
>> Hmmmmmm:  media_errors = 0   Yippee!
>>
>
> Welcome :-)
>
> Now we have to find if there is a Windows version ;-)
>

I use gsmartcontrol on Windows

Stan Brown <the_stan_brown@fastmail.fm> wrote:
> There doesn't seem to be an easy answer to my query about not having
> Robocopy write files when the source and destination have identical
> contents but different timestamps. The main reason I cared was not
> wanting to wear out flash memory prematurely.
>
> That leads me to ask: is there any way to tell when a flash drive is
> getting close to needing replacement, before it actually starts to
> accumulate data errors?

Not an answer to your question, but perhaps to your - implied -
concern:

As part of my backup procedures, I've been making daily backup of my
most important files to 'flash' for about two decades. First to SD
memory cards, now (for over a year) to USB memory sticks.

In all this time, so well over 7000 backups, mostly writing over the
same locations of the media, I've not had a single failure, not during
backup/writing and not during reading (for subsequent backups to other
media).

The only failure I had, was a totally dead USB memory stick, which was
hardly ever used (probably only a few times). That stick was not a real
brand, only the brand of a Dutch drugstore chain. (I only mention this
one failure as to paint a complete story, not a rosy one).

This of course don't tell you how the media is going to fail, nor if
there will be any early warning signs. It's just an indication that -
will reputable brand media - the media is not likely gooing to fail.

And perhaps you can add a read-after-write part to your procedure,
similar to what I do.

On 10/27/2023 10:20 AM, Frank Slootweg wrote:
> Stan Brown <the_stan_brown@fastmail.fm> wrote:
>> There doesn't seem to be an easy answer to my query about not having
>> Robocopy write files when the source and destination have identical
>> contents but different timestamps. The main reason I cared was not
>> wanting to wear out flash memory prematurely.
>>
>> That leads me to ask: is there any way to tell when a flash drive is
>> getting close to needing replacement, before it actually starts to
>> accumulate data errors?
>
> Not an answer to your question, but perhaps to your - implied -
> concern:
>
> As part of my backup procedures, I've been making daily backup of my
> most important files to 'flash' for about two decades. First to SD
> memory cards, now (for over a year) to USB memory sticks.
>
> In all this time, so well over 7000 backups, mostly writing over the
> same locations of the media, I've not had a single failure, not during
> backup/writing and not during reading (for subsequent backups to other
> media).
>
> The only failure I had, was a totally dead USB memory stick, which was
> hardly ever used (probably only a few times). That stick was not a real
> brand, only the brand of a Dutch drugstore chain. (I only mention this
> one failure as to paint a complete story, not a rosy one).
>
> This of course don't tell you how the media is going to fail, nor if
> there will be any early warning signs. It's just an indication that -
> will reputable brand media - the media is not likely gooing to fail.
>
> And perhaps you can add a read-after-write part to your procedure,
> similar to what I do.
>

What capacity does each USB stick have ?

Paul

Paul <nospam@needed.invalid> wrote:
> On 10/27/2023 10:20 AM, Frank Slootweg wrote:
> > Stan Brown <the_stan_brown@fastmail.fm> wrote:
> >> There doesn't seem to be an easy answer to my query about not having
> >> Robocopy write files when the source and destination have identical
> >> contents but different timestamps. The main reason I cared was not
> >> wanting to wear out flash memory prematurely.
> >>
> >> That leads me to ask: is there any way to tell when a flash drive is
> >> getting close to needing replacement, before it actually starts to
> >> accumulate data errors?
> >
> > Not an answer to your question, but perhaps to your - implied -
> > concern:
> >
> > As part of my backup procedures, I've been making daily backup of my
> > most important files to 'flash' for about two decades. First to SD
> > memory cards, now (for over a year) to USB memory sticks.
> >
> > In all this time, so well over 7000 backups, mostly writing over the
> > same locations of the media, I've not had a single failure, not during
> > backup/writing and not during reading (for subsequent backups to other
> > media).
> >
> > The only failure I had, was a totally dead USB memory stick, which was
> > hardly ever used (probably only a few times). That stick was not a real
> > brand, only the brand of a Dutch drugstore chain. (I only mention this
> > one failure as to paint a complete story, not a rosy one).
> >
> > This of course don't tell you how the media is going to fail, nor if
> > there will be any early warning signs. It's just an indication that -
> > will reputable brand media - the media is not likely gooing to fail.
> >
> > And perhaps you can add a read-after-write part to your procedure,
> > similar to what I do.
>
> What capacity does each USB stick have ?

The SD memory cards started at 1GB (remember, two decades ago) and the
last one was 4GB. The current USB memory stick is 64GB, but with only
some 1.2GB 'Used space'. It was the smallest (in GB), good brand, good
physical quality, small physical size, one I could buy (Samsung Fit Plus).

On 27 Oct 2023 14:20:19 GMT, Frank Slootweg wrote:
> In all this time, so well over 7000 backups, mostly writing over the
> same locations of the media, I've not had a single failure, not during
> backup/writing and not during reading (for subsequent backups to other
> media).
>
> The only failure I had, was a totally dead USB memory stick, which was
> hardly ever used (probably only a few times). That stick was not a real
> brand, only the brand of a Dutch drugstore chain. (I only mention this
> one failure as to paint a complete story, not a rosy one).
>
> This of course don't tell you how the media is going to fail, nor if
> there will be any early warning signs. It's just an indication that -
> will reputable brand media - the media is not likely gooing to fail.
>
> And perhaps you can add a read-after-write part to your procedure,
> similar to what I do.

Mine's a 256 GB SanDisk, with dual connectors to USB-A
(for desktop PC) and USB-C (for laptop). About 2 GB is
used for my files.

I use Robocopy, and I thought it checked the integrity
of copy files, but in skimming the user manual I don't
see anything to confirm that. Does anyone know for
sure?

--
Stan Brown, Tehachapi, California, USA
https://BrownMath.com/
Shikata ga nai...

On 10/28/2023 2:41 PM, Stan Brown wrote:
> On 27 Oct 2023 14:20:19 GMT, Frank Slootweg wrote:
>> In all this time, so well over 7000 backups, mostly writing over the
>> same locations of the media, I've not had a single failure, not during
>> backup/writing and not during reading (for subsequent backups to other
>> media).
>>
>> The only failure I had, was a totally dead USB memory stick, which was
>> hardly ever used (probably only a few times). That stick was not a real
>> brand, only the brand of a Dutch drugstore chain. (I only mention this
>> one failure as to paint a complete story, not a rosy one).
>>
>> This of course don't tell you how the media is going to fail, nor if
>> there will be any early warning signs. It's just an indication that -
>> will reputable brand media - the media is not likely gooing to fail.
>>
>> And perhaps you can add a read-after-write part to your procedure,
>> similar to what I do.
>
>
> Mine's a 256 GB SanDisk, with dual connectors to USB-A
> (for desktop PC) and USB-C (for laptop). About 2 GB is
> used for my files.
>
> I use Robocopy, and I thought it checked the integrity
> of copy files, but in skimming the user manual I don't
> see anything to confirm that. Does anyone know for
> sure?
>

Robocopy is not a synchronization utility.

It is a fancy copy utility with pseudo-policies of some sort (MIR).

Some functions in life, are outside of the responsibilities of a copy utility.
Then you need a synchronization utility.

*******

Since robocopy can create a logfile of what was done, if you
need meat to feed into a verification routine, that would be
a way of getting the materials. The Robocopy runs I've done in
the past, I always generated a logfile for each session. "Just in case".
It may state in there, why a particular file was skipped.

Paul

On Sat, 28 Oct 2023 17:01:16 -0400, Paul wrote:
> Robocopy is not a synchronization utility.
>
> It is a fancy copy utility with pseudo-policies of some sort (MIR).
>
> Some functions in life, are outside of the responsibilities of a copy utility.
> Then you need a synchronization utility.

You've lost me. What features would be in a sync utility that aren't
in Robocopy?

--
Stan Brown, Tehachapi, California, USA https://BrownMath.com/
Shikata ga nai...

On 10/29/2023 6:02 PM, Stan Brown wrote:
> On Sat, 28 Oct 2023 17:01:16 -0400, Paul wrote:
>> Robocopy is not a synchronization utility.
>>
>> It is a fancy copy utility with pseudo-policies of some sort (MIR).
>>
>> Some functions in life, are outside of the responsibilities of a copy utility.
>> Then you need a synchronization utility.
>
> You've lost me. What features would be in a sync utility that aren't
> in Robocopy?
>

You asked for a scheme, which did not use the date, but
referred to content changes as the trigger for copying
the new info.

A synchronization scheme should not "rely" on the date,
since date stamps on a computer are not reliable. For
example, TZ and FAT32 mess with the details (your portable device
may have been used on multiple computers, ones with messed up
clocks or whatever - not all the computers in the house have
valid TZ configurations). A synchronization program then, could have
a vested interest in examining content as a trigger. Perhaps
there would be a tick box for that.

A synchronization program might have to ask you "what should I do?",
when an unexpected thing happens. Robocopy isn't designed for
stopping particularly. It will "skip" stuff not meeting requirements
and make a log entry. This is why you always generate a logfile
when running Robocopy (and walking away from it).

This is outside the purview of simple copy programs such as Robocopy.
Robocopy approach is prefaced on ordinary rotating hard drives,
where there is no appreciable wear from doing excess writes.

Normally, with hard drives, there is no cost other than time wasted,
for copying a file you suspect may be more recent than the one
at the destination. However, on flash based devices, you have the
"wear aspect" to worry about. Then, it is a user tradeoff, as
to whether writes are to be avoided, by comparing file contents (slower)
to see if doing a write is really necessary. A further optimization,
would only be updating blocks that had changed. Not all file formats
make that a useful approach. Only some, change at the individual block
level.

The author of Robocopy, thought he was doing a replacement
for xxcopy, rather than a feature set for file synchronization
(where files could travel in both directions, not just one direction).

I know some concepts, but I don't use all these ideas in practice.
(Any sync I was doing, was done manually, between home and work.)

You can see some other Microsoft projects here. There was never a plan
for these developer-inspired private projects, to make it into the OS.
That's what is so unusual about Robocopy -- it actually earned a home
in the OS and is maintained. The Powertoys might have some "support",
but they're a separate download.

https://en.wikipedia.org/wiki/RichCopy (like Robocopy, except GUI based)

https://en.wikipedia.org/wiki/SyncToy (still doesn't analyze content, uses dates)

Paul

On Sun, 29 Oct 2023 19:25:49 -0400, Paul wrote:
>
> On 10/29/2023 6:02 PM, Stan Brown wrote:
> > On Sat, 28 Oct 2023 17:01:16 -0400, Paul wrote:
> >> Robocopy is not a synchronization utility.
> >>
> >> It is a fancy copy utility with pseudo-policies of some sort (MIR).
> >>
> >> Some functions in life, are outside of the responsibilities of a copy utility.
> >> Then you need a synchronization utility.
> >
> > You've lost me. What features would be in a sync utility that aren't
> > in Robocopy?
>
> You asked for a scheme, which did not use the date, but
> referred to content changes as the trigger for copying
> the new info.

Not
> A synchronization scheme should not "rely" on the date,
> since date stamps on a computer are not reliable. For
> example, TZ and FAT32 mess with the details (your portable device
> may have been used on multiple computers, ones with messed up
> clocks or whatever - not all the computers in the house have
> valid TZ configurations). A synchronization program then, could have
> a vested interest in examining content as a trigger. Perhaps
> there would be a tick box for that.
>
> A synchronization program might have to ask you "what should I do?",
> when an unexpected thing happens. Robocopy isn't designed for
> stopping particularly. It will "skip" stuff not meeting requirements
> and make a log entry. This is why you always generate a logfile
> when running Robocopy (and walking away from it).
>
> This is outside the purview of simple copy programs such as Robocopy.
> Robocopy approach is prefaced on ordinary rotating hard drives,
> where there is no appreciable wear from doing excess writes.
>
> Normally, with hard drives, there is no cost other than time wasted,
> for copying a file you suspect may be more recent than the one
> at the destination. However, on flash based devices, you have the
> "wear aspect" to worry about. Then, it is a user tradeoff, as
> to whether writes are to be avoided, by comparing file contents (slower)
> to see if doing a write is really necessary. A further optimization,
> would only be updating blocks that had changed. Not all file formats
> make that a useful approach. Only some, change at the individual block
> level.
>
> The author of Robocopy, thought he was doing a replacement
> for xxcopy, rather than a feature set for file synchronization
> (where files could travel in both directions, not just one direction).

If I understand you correctly, then I do want a file
copy utility, not a synchronization utility. While
files travel in two directions in different runs, in
any one run they travel in only one direction.

Here's my workflow:

Work on stuff on desktop PC
Files copy desktop --> USB stick
Files copy USB stick --> laptop
Work on stuff on laptop
Files copy laptop --> USB stick
Files copy USB stick --> desktop PC
And repeat

Typically not more than a few MB get copied in any one
run.
> I know some concepts, but I don't use all these ideas in practice.
> (Any sync I was doing, was done manually, between home and work.)

--
Stan Brown, Tehachapi, California, USA
https://BrownMath.com/
Shikata ga nai...

On Mon, 30 Oct 2023 07:07:16 -0700, Stan Brown <the_stan_brown@fastmail.fm>
wrote:

>Here's my workflow:

Seeing someone else's workflow causes me to think about my own approach.
- Place files in shared folder(s)
- Work on stuff from anywhere

>Work on stuff on desktop PC
>Files copy desktop --> USB stick
>Files copy USB stick --> laptop
>Work on stuff on laptop
>Files copy laptop --> USB stick
>Files copy USB stick --> desktop PC
>And repeat
>
>Typically not more than a few MB get copied in any one
>run.

Stan Brown <the_stan_brown@fastmail.fm> wrote:
[...]

> If I understand you correctly, then I do want a file
> copy utility, not a synchronization utility. While
> files travel in two directions in different runs, in
> any one run they travel in only one direction.
>
> Here's my workflow:
>
> Work on stuff on desktop PC
> Files copy desktop --> USB stick
> Files copy USB stick --> laptop
> Work on stuff on laptop
> Files copy laptop --> USB stick
> Files copy USB stick --> desktop PC
> And repeat
>
> Typically not more than a few MB get copied in any one
> run.

Pardon my ignorance, but can't you just use XCOPY with the '/A' and '/M'
options?

/A Copies only files with the archive attribute set,
doesn't change the attribute.
/M Copies only files with the archive attribute set,
turns off the archive attribute.

It seems that the USB stick is your 'archive', and the desktop PC and
laptop just have copies of that archive (and work-in-progress files).

So this is how it would work:

Work on stuff on desktop PC. Archive attribute is set *only* on
*modified*/*new* files.

'XCOPY /A' copies *only* modified/new files to USB stick.
Archive attributes unchanged.
So the wear-and-tear of the USB stick is minimized, exactly like you
want.

'XCOPY /M' copies *only* modified/new files to laptop.
Archive attributes are turned off.

At that moment, the desktop PC, USB stick and laptop all have exactly
the same files.

For the reverse, laptop via USB stick to desktop PC, again use
'XCOPY /A' for the transfer to the USB stick and 'XCOPY /M' for the
transfer from the USB stick.

N.B. Of course this procedure (probably) only works after an initial
set of 'full' transfers, because (probably) the archive attribute is
still set on all/most files (unless cleared by some other backup
mechanism).

Bottom line: This is exactly why the mechanism of the archive
attribute - the 'A-bit' - was designed the way it was.

HTH.