Has any body an explanation why when I copy 40 gb of video files to a
usb hard drive that is plugged in to a powered usb hub takes over an
hour to copy while using a software called FreeFileSync only takes 5
minutes.

I am using NEMO as my file manager to copy from one drive to another. I
suspect that my file manager may not be a good way to copy large
amounts of files.

On 1/24/24 01:14 PM, Nic wrote:
> Has any body an explanation why when I copy 40 gb of video files to a
> usb hard drive that is plugged in to a powered usb hub takes over an
> hour to copy while using a software called FreeFileSync only takes 5
> minutes.
>
> I am using NEMO as my file manager to copy from one drive to another. I
> suspect that my file manager may not be a good way to copy large
> amounts of files.

File sync programs usually (not always) attempt to skip files that do
not need to be updated. Saving time.

I user rsync with a few parameters to do my updates. And it does just
as I described, saving time when I update my MP3 player. And yes, I
still have an MP3 player with earphones LOL.
--
Linux Mint 21.2 Cinnamon
Al

On 1/24/24 11:14 AM, Nic wrote:
> Has any body an explanation why when I copy 40 gb of video files to a
> usb hard drive that is plugged in to a powered usb hub takes over an
> hour to copy while using a software called FreeFileSync only takes 5
> minutes.
>
> I am using NEMO as my file manager to copy from one drive to another. I
> suspect that my file manager may not be a good way to copy large
> amounts of files.
>

Nemo is just doing straight copies while "Free"FileSync is doing diffs
with a previously populated target drive. I'd think the *initial*
population of the target drive would take about the same amount of time
since *all* files are being copied, not just the diffs.

Regarding USB mounted memory storage my experience is there can be a
great deal of variation is transfer speeds between different brands.
And of course having USB 3.1 or 3.2 helps.

You could probably match or beat the performance of "Free"FileSync by
using rsync or similar archiving scheme.

On 1/24/24 1:50 PM, Big Al wrote:
> On 1/24/24 01:14 PM, Nic wrote:
>> Has any body an explanation why when I copy 40 gb of video files to a
>> usb hard drive that is plugged in to a powered usb hub takes over an
>> hour to copy while using a software called FreeFileSync only takes 5
>> minutes.
>>
>> I am using NEMO as my file manager to copy from one drive to another. I
>> suspect that my file manager may not be a good way to copy large
>> amounts of files.
>
> File sync programs usually (not always) attempt to skip files that do
> not need to be updated.  Saving time.
>
> I user rsync with a few parameters to do my updates.  And it does just
> as I described, saving time when I update my MP3 player.  And yes, I
> still have an MP3 player with earphones LOL.
You do not understand my question.

On 1/24/24 1:52 PM, Arti F. Idiot wrote:
> On 1/24/24 11:14 AM, Nic wrote:
>> Has any body an explanation why when I copy 40 gb of video files to a
>> usb hard drive that is plugged in to a powered usb hub takes over an
>> hour to copy while using a software called FreeFileSync only takes 5
>> minutes.
>>
>> I am using NEMO as my file manager to copy from one drive to another. I
>> suspect that my file manager may not be a good way to copy large
>> amounts of files.
>>
>
> Nemo is just doing straight copies while "Free"FileSync is doing diffs
> with a previously populated target drive.  I'd think the *initial*
> population of the target drive would take about the same amount of
> time since *all* files are being copied, not just the diffs.
>
> Regarding USB mounted memory storage my experience is there can be a
> great deal of variation is transfer speeds between different brands.
> And of course having USB 3.1 or 3.2 helps.
>
> You could probably match or beat the performance of "Free"FileSync by
> using rsync or similar archiving scheme.
I am simply copying from one drive to another. It seems that
FreeFileSync is using a different method to copy the files than NEMO is
using.

On 1/24/24 01:58 PM, Nic wrote:
> On 1/24/24 1:50 PM, Big Al wrote:
>> On 1/24/24 01:14 PM, Nic wrote:
>>> Has any body an explanation why when I copy 40 gb of video files to a
>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>> hour to copy while using a software called FreeFileSync only takes 5
>>> minutes.
>>>
>>> I am using NEMO as my file manager to copy from one drive to another. I
>>> suspect that my file manager may not be a good way to copy large
>>> amounts of files.
>>
>> File sync programs usually (not always) attempt to skip files that do
>> not need to be updated.  Saving time.
>>
>> I user rsync with a few parameters to do my updates.  And it does just
>> as I described, saving time when I update my MP3 player.  And yes, I
>> still have an MP3 player with earphones LOL.
> You do not understand my question.
I do, you wonder why freefilesync works faster! I explained a possible
answer. I do not know/use freefilesync so I'm not 100% sure.
--
Linux Mint 21.2 Cinnamon
Al

On 1/24/24 02:00 PM, Nic wrote:
> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>> On 1/24/24 11:14 AM, Nic wrote:
>>> Has any body an explanation why when I copy 40 gb of video files to a
>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>> hour to copy while using a software called FreeFileSync only takes 5
>>> minutes.
>>>
>>> I am using NEMO as my file manager to copy from one drive to another. I
>>> suspect that my file manager may not be a good way to copy large
>>> amounts of files.
>>>
>>
>> Nemo is just doing straight copies while "Free"FileSync is doing diffs
>> with a previously populated target drive.  I'd think the *initial*
>> population of the target drive would take about the same amount of
>> time since *all* files are being copied, not just the diffs.
>>
>> Regarding USB mounted memory storage my experience is there can be a
>> great deal of variation is transfer speeds between different brands.
>> And of course having USB 3.1 or 3.2 helps.
>>
>> You could probably match or beat the performance of "Free"FileSync by
>> using rsync or similar archiving scheme.
> I am simply copying from one drive to another. It seems that
> FreeFileSync is using a different method to copy the files than NEMO is
> using.
And that's the same thing I said. Arti and I are trying to say that
FreeFileSync is intelligent. Nemo is dumb.
--
Linux Mint 21.2 Cinnamon
Al

On 1/24/24 12:00 PM, Nic wrote:
> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>> On 1/24/24 11:14 AM, Nic wrote:
>>> Has any body an explanation why when I copy 40 gb of video files to a
>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>> hour to copy while using a software called FreeFileSync only takes 5
>>> minutes.
>>>
>>> I am using NEMO as my file manager to copy from one drive to another. I
>>> suspect that my file manager may not be a good way to copy large
>>> amounts of files.
>>>
>>
>> Nemo is just doing straight copies while "Free"FileSync is doing diffs
>> with a previously populated target drive.  I'd think the *initial*
>> population of the target drive would take about the same amount of
>> time since *all* files are being copied, not just the diffs.
>>
>> Regarding USB mounted memory storage my experience is there can be a
>> great deal of variation is transfer speeds between different brands.
>> And of course having USB 3.1 or 3.2 helps.
>>
>> You could probably match or beat the performance of "Free"FileSync by
>> using rsync or similar archiving scheme.
> I am simply copying from one drive to another. It seems that
> FreeFileSync is using a different method to copy the files than NEMO is
> using.

So the USB drive in question is completely empty, devoid of prior files?
I think what you're saying then doesn't make much sense because there
are upper limits to how fast data can be locally moved over the bus.
Perhaps it only seems like it's faster because your stuff is really on
some cloud?

On 1/24/24 2:37 PM, Big Al wrote:
> On 1/24/24 01:58 PM, Nic wrote:
>> On 1/24/24 1:50 PM, Big Al wrote:
>>> On 1/24/24 01:14 PM, Nic wrote:
>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>> minutes.
>>>>
>>>> I am using NEMO as my file manager to copy from one drive to
>>>> another. I
>>>> suspect that my file manager may not be a good way to copy large
>>>> amounts of files.
>>>
>>> File sync programs usually (not always) attempt to skip files that
>>> do not need to be updated.  Saving time.
>>>
>>> I user rsync with a few parameters to do my updates.  And it does
>>> just as I described, saving time when I update my MP3 player.  And
>>> yes, I still have an MP3 player with earphones LOL.
>> You do not understand my question.
> I do, you wonder why freefilesync works faster!   I explained a
> possible answer.  I do not know/use freefilesync so I'm not 100% sure.
There is no updating involved, just copy to another drive. The files I
am copying are going on to the other drive for the first time, a clean
copy not updating.

On 1/24/24 2:38 PM, Big Al wrote:
> On 1/24/24 02:00 PM, Nic wrote:
>> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>>> On 1/24/24 11:14 AM, Nic wrote:
>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>> minutes.
>>>>
>>>> I am using NEMO as my file manager to copy from one drive to
>>>> another. I
>>>> suspect that my file manager may not be a good way to copy large
>>>> amounts of files.
>>>>
>>>
>>> Nemo is just doing straight copies while "Free"FileSync is doing
>>> diffs with a previously populated target drive.  I'd think the
>>> *initial* population of the target drive would take about the same
>>> amount of time since *all* files are being copied, not just the diffs.
>>>
>>> Regarding USB mounted memory storage my experience is there can be a
>>> great deal of variation is transfer speeds between different brands.
>>> And of course having USB 3.1 or 3.2 helps.
>>>
>>> You could probably match or beat the performance of "Free"FileSync
>>> by using rsync or similar archiving scheme.
>> I am simply copying from one drive to another. It seems that
>> FreeFileSync is using a different method to copy the files than NEMO
>> is using.
> And that's the same thing I said.  Arti and I are trying to say that
> FreeFileSync is intelligent.  Nemo is dumb.
It seems that way.

On 1/24/24 2:40 PM, Arti F. Idiot wrote:
> On 1/24/24 12:00 PM, Nic wrote:
>> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>>> On 1/24/24 11:14 AM, Nic wrote:
>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>> minutes.
>>>>
>>>> I am using NEMO as my file manager to copy from one drive to
>>>> another. I
>>>> suspect that my file manager may not be a good way to copy large
>>>> amounts of files.
>>>>
>>>
>>> Nemo is just doing straight copies while "Free"FileSync is doing
>>> diffs with a previously populated target drive.  I'd think the
>>> *initial* population of the target drive would take about the same
>>> amount of time since *all* files are being copied, not just the diffs.
>>>
>>> Regarding USB mounted memory storage my experience is there can be a
>>> great deal of variation is transfer speeds between different brands.
>>> And of course having USB 3.1 or 3.2 helps.
>>>
>>> You could probably match or beat the performance of "Free"FileSync
>>> by using rsync or similar archiving scheme.
>> I am simply copying from one drive to another. It seems that
>> FreeFileSync is using a different method to copy the files than NEMO
>> is using.
>
> So the USB drive in question is completely empty, devoid of prior
> files?  I think what you're saying then doesn't make much sense
> because there are upper limits to how fast data can be locally moved
> over the bus. Perhaps it only seems like it's faster because your
> stuff is really on some cloud?
No cloud, just local files from my secondary drive to a usb drive. Nemo
gets a failing mark for copying large  80gb and upwards file sizes.

On 1/24/24 02:48 PM, Nic wrote:
> On 1/24/24 2:37 PM, Big Al wrote:
>> On 1/24/24 01:58 PM, Nic wrote:
>>> On 1/24/24 1:50 PM, Big Al wrote:
>>>> On 1/24/24 01:14 PM, Nic wrote:
>>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>>> minutes.
>>>>>
>>>>> I am using NEMO as my file manager to copy from one drive to
>>>>> another. I
>>>>> suspect that my file manager may not be a good way to copy large
>>>>> amounts of files.
>>>>
>>>> File sync programs usually (not always) attempt to skip files that
>>>> do not need to be updated.  Saving time.
>>>>
>>>> I user rsync with a few parameters to do my updates.  And it does
>>>> just as I described, saving time when I update my MP3 player.  And
>>>> yes, I still have an MP3 player with earphones LOL.
>>> You do not understand my question.
>> I do, you wonder why freefilesync works faster!   I explained a
>> possible answer.  I do not know/use freefilesync so I'm not 100% sure.
> There is no updating involved, just copy to another drive. The files I
> am copying are going on to the other drive for the first time, a clean
> copy not updating.
That would have been good info in the OP
--
Linux Mint 21.2 Cinnamon
Al

On 2024-01-24, Big Al <alan@invalid.com> wrote:
> On 1/24/24 02:00 PM, Nic wrote:
>> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>>> On 1/24/24 11:14 AM, Nic wrote:
>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>> minutes.
>>>>
>>>> I am using NEMO as my file manager to copy from one drive to another. I
>>>> suspect that my file manager may not be a good way to copy large
>>>> amounts of files.
>>>>
>>>
>>> Nemo is just doing straight copies while "Free"FileSync is doing diffs
>>> with a previously populated target drive.  I'd think the *initial*
>>> population of the target drive would take about the same amount of
>>> time since *all* files are being copied, not just the diffs.
>>>
>>> Regarding USB mounted memory storage my experience is there can be a
>>> great deal of variation is transfer speeds between different brands.
>>> And of course having USB 3.1 or 3.2 helps.
>>>
>>> You could probably match or beat the performance of "Free"FileSync by
>>> using rsync or similar archiving scheme.
>> I am simply copying from one drive to another. It seems that
>> FreeFileSync is using a different method to copy the files than NEMO is
>> using.
> And that's the same thing I said. Arti and I are trying to say that
> FreeFileSync is intelligent. Nemo is dumb.

Correct. Another way of looking at it is, that copy and paste is not not the
correct tool to use. Copy and paste is really for small files.

Rsync's function is to copy (and move) files, big small and lots of them.

Grsync is another front end for rsync.

On 2024-01-24, Nic <Nic@none.invalid> wrote:
> On 1/24/24 2:38 PM, Big Al wrote:
>> On 1/24/24 02:00 PM, Nic wrote:
>>> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>>>> On 1/24/24 11:14 AM, Nic wrote:
>>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>>> minutes.
>>>>>
>>>>> I am using NEMO as my file manager to copy from one drive to
>>>>> another. I
>>>>> suspect that my file manager may not be a good way to copy large
>>>>> amounts of files.
>>>>>
>>>>
>>>> Nemo is just doing straight copies while "Free"FileSync is doing
>>>> diffs with a previously populated target drive.  I'd think the
>>>> *initial* population of the target drive would take about the same
>>>> amount of time since *all* files are being copied, not just the diffs.
>>>>
>>>> Regarding USB mounted memory storage my experience is there can be a
>>>> great deal of variation is transfer speeds between different brands.
>>>> And of course having USB 3.1 or 3.2 helps.
>>>>
>>>> You could probably match or beat the performance of "Free"FileSync
>>>> by using rsync or similar archiving scheme.
>>> I am simply copying from one drive to another. It seems that
>>> FreeFileSync is using a different method to copy the files than NEMO
>>> is using.
>> And that's the same thing I said.  Arti and I are trying to say that
>> FreeFileSync is intelligent.  Nemo is dumb.
> It seems that way.

The fist review on the FreeFileSynce web site says

"Excellent, efficient tool for manual backup and folder mirroring. No
persistent services. Does what it promises and is updated frequently.
Takes advantage of multicore now. A big thanks to the developers."

Not the use of mutli core.

On 1/24/24 3:53 PM, Gordon wrote:
> On 2024-01-24, Big Al <alan@invalid.com> wrote:
>> On 1/24/24 02:00 PM, Nic wrote:
>>> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>>>> On 1/24/24 11:14 AM, Nic wrote:
>>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>>> minutes.
>>>>>
>>>>> I am using NEMO as my file manager to copy from one drive to another. I
>>>>> suspect that my file manager may not be a good way to copy large
>>>>> amounts of files.
>>>>>
>>>> Nemo is just doing straight copies while "Free"FileSync is doing diffs
>>>> with a previously populated target drive.  I'd think the *initial*
>>>> population of the target drive would take about the same amount of
>>>> time since *all* files are being copied, not just the diffs.
>>>>
>>>> Regarding USB mounted memory storage my experience is there can be a
>>>> great deal of variation is transfer speeds between different brands.
>>>> And of course having USB 3.1 or 3.2 helps.
>>>>
>>>> You could probably match or beat the performance of "Free"FileSync by
>>>> using rsync or similar archiving scheme.
>>> I am simply copying from one drive to another. It seems that
>>> FreeFileSync is using a different method to copy the files than NEMO is
>>> using.
>> And that's the same thing I said. Arti and I are trying to say that
>> FreeFileSync is intelligent. Nemo is dumb.
> Correct. Another way of looking at it is, that copy and paste is not not the
> correct tool to use. Copy and paste is really for small files.
>
> Rsync's function is to copy (and move) files, big small and lots of them.
>
> Grsync is another front end for rsync.
Thank you. Installed the front end and is working nicely. Pity about
Nemo, is there a smarter filemanager?

On 1/24/24 05:14 PM, Nic wrote:
> On 1/24/24 3:53 PM, Gordon wrote:
>> On 2024-01-24, Big Al <alan@invalid.com> wrote:
>>> On 1/24/24 02:00 PM, Nic wrote:
>>>> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>>>>> On 1/24/24 11:14 AM, Nic wrote:
>>>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>>>> minutes.
>>>>>>
>>>>>> I am using NEMO as my file manager to copy from one drive to
>>>>>> another. I
>>>>>> suspect that my file manager may not be a good way to copy large
>>>>>> amounts of files.
>>>>>>
>>>>> Nemo is just doing straight copies while "Free"FileSync is doing diffs
>>>>> with a previously populated target drive.  I'd think the *initial*
>>>>> population of the target drive would take about the same amount of
>>>>> time since *all* files are being copied, not just the diffs.
>>>>>
>>>>> Regarding USB mounted memory storage my experience is there can be a
>>>>> great deal of variation is transfer speeds between different brands.
>>>>> And of course having USB 3.1 or 3.2 helps.
>>>>>
>>>>> You could probably match or beat the performance of "Free"FileSync by
>>>>> using rsync or similar archiving scheme.
>>>> I am simply copying from one drive to another. It seems that
>>>> FreeFileSync is using a different method to copy the files than NEMO is
>>>> using.
>>> And that's the same thing I said.  Arti and I are trying to say that
>>> FreeFileSync is intelligent.  Nemo is dumb.
>> Correct. Another way of looking at it is, that copy and paste is not
>> not the
>> correct tool to use. Copy and paste is really for small files.
>>
>> Rsync's function is to copy (and move) files, big small and lots of them.
>>
>> Grsync is another front end for rsync.
> Thank you. Installed the front end and is working nicely. Pity about
> Nemo, is there a smarter filemanager?
In my system 21.2 Nautilus is installed already.
Can't tell you how to make it work other than terminal and type
'nautilus' <no quotes>
--
Linux Mint 21.2 Cinnamon
Al

On 1/24/24 06:33 PM, Big Al wrote:
> On 1/24/24 05:14 PM, Nic wrote:
>> On 1/24/24 3:53 PM, Gordon wrote:
>>> On 2024-01-24, Big Al <alan@invalid.com> wrote:
>>>> On 1/24/24 02:00 PM, Nic wrote:
>>>>> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
>>>>>> On 1/24/24 11:14 AM, Nic wrote:
>>>>>>> Has any body an explanation why when I copy 40 gb of video files
>>>>>>> to a
>>>>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>>>>> minutes.
>>>>>>>
>>>>>>> I am using NEMO as my file manager to copy from one drive to
>>>>>>> another. I
>>>>>>> suspect that my file manager may not be a good way to copy large
>>>>>>> amounts of files.
>>>>>>>
>>>>>> Nemo is just doing straight copies while "Free"FileSync is doing
>>>>>> diffs
>>>>>> with a previously populated target drive.  I'd think the *initial*
>>>>>> population of the target drive would take about the same amount of
>>>>>> time since *all* files are being copied, not just the diffs.
>>>>>>
>>>>>> Regarding USB mounted memory storage my experience is there can be a
>>>>>> great deal of variation is transfer speeds between different brands.
>>>>>> And of course having USB 3.1 or 3.2 helps.
>>>>>>
>>>>>> You could probably match or beat the performance of "Free"FileSync by
>>>>>> using rsync or similar archiving scheme.
>>>>> I am simply copying from one drive to another. It seems that
>>>>> FreeFileSync is using a different method to copy the files than
>>>>> NEMO is
>>>>> using.
>>>> And that's the same thing I said.  Arti and I are trying to say that
>>>> FreeFileSync is intelligent.  Nemo is dumb.
>>> Correct. Another way of looking at it is, that copy and paste is not
>>> not the
>>> correct tool to use. Copy and paste is really for small files.
>>>
>>> Rsync's function is to copy (and move) files, big small and lots of
>>> them.
>>>
>>> Grsync is another front end for rsync.
>> Thank you. Installed the front end and is working nicely. Pity about
>> Nemo, is there a smarter filemanager?
> In my system 21.2 Nautilus is installed already.
> Can't tell you how to make it work other than terminal and type
> 'nautilus' <no quotes>
And can't tell you it's any better, probably not.
--
Linux Mint 21.2 Cinnamon
Al

On 1/24/2024 2:48 PM, Nic wrote:
> On 1/24/24 2:37 PM, Big Al wrote:
>> On 1/24/24 01:58 PM, Nic wrote:
>>> On 1/24/24 1:50 PM, Big Al wrote:
>>>> On 1/24/24 01:14 PM, Nic wrote:
>>>>> Has any body an explanation why when I copy 40 gb of video files to a
>>>>> usb hard drive that is plugged in to a powered usb hub takes over an
>>>>> hour to copy while using a software called FreeFileSync only takes 5
>>>>> minutes.
>>>>>
>>>>> I am using NEMO as my file manager to copy from one drive to another. I
>>>>> suspect that my file manager may not be a good way to copy large
>>>>> amounts of files.
>>>>
>>>> File sync programs usually (not always) attempt to skip files that do not need to be updated.  Saving time.
>>>>
>>>> I user rsync with a few parameters to do my updates.  And it does just as I described, saving time when I update my MP3 player.  And yes, I still have an MP3 player with earphones LOL.
>>> You do not understand my question.
>> I do, you wonder why freefilesync works faster!   I explained a possible answer.  I do not know/use freefilesync so I'm not 100% sure.
> There is no updating involved, just copy to another drive. The files I am copying are going on to the other drive for the first time, a clean copy not updating.

FreeFileSync is multithreaded, which may be an
advantage if both the source drive and the
destination drive are flash based.

One other claim, is that from the GUI, the
program does not carry out activity until
commanded to do so. This means it is not
necessarily cheating, by reading files
while you are thinking about what to sync.

Using multithreading on a HDD, causes it to
thrash to some extent. But that's subject a
bit, to how the drive is fitted with cache
and AHCI (or the kind of AHCI-like behavior available
on a SAS drive, where the max tags is 64K).

For example, a HDD sitting on a real RAID hardware
controller with a couple gigabytes of DRAM cache, might
behave a bit better. You never know.

Now you could give us hardware details about what
your two devices were, to make it easier to understand
how this was possible. For example, if I knew in advance
that both the source and destination were rotating media
drives, that would make it a lot harder to explain.

Paul

On 1/24/2024 2:50 PM, Nic wrote:

> Nemo gets a failing mark for copying large  80gb and upwards file sizes.

https://en.wikipedia.org/wiki/Nemo_%28file_manager%29

"Nemo... is a fork of GNOME Files (formerly named Nautilus)."

"It started... after the developers of the operating system Linux Mint
considered that "Nautilus 3.6 is a catastrophe"
"

The thread here has a bit of whining and moaning,
but what they're not discussing is Linux has a write cache,
and they don't seem to be taking any precautions.

https://forums.linuxmint.com/viewtopic.php?t=348708

Having a "Safely Remove" feature for drive unmounting
and removal, instead of just "yanking out the cable",
will result in a lot more file integrity.

For example, if a device has a GUI option called "eject",
that implies the software will do all of the necessary
things for safe usage.

One wag claims to be "hashing the source and dest file
and seeing they are the same, yet the file is corrupted".
Well, of course, he is hashing the contents of the write
cache and not the contents of the disk, while a transfer
is in-flight. At least one user mentions "sync" and that
person would be from the Unix era and know about
flushing by using the sync command multiple times :-)

"The 'sync' Command In Linux RobertElderSoftware"

https://www.youtube.com/watch?v=cq3fgnJMC8c

The manual page is not all that helpful.

https://linux.die.net/man/8/sync

*******

The graphics in that video are pretty bad, but what the
guy wants you to watch is:

1) You have two drives.
2) Each drive has a LED on it, for activity.
3)
cp /media/bullwinkle/drive1/big.file /media/bullwinkle/drive2/
# The command has come back. Yet, the transfer is *not* complete.
# Watch the LED on the destination drive.
# OK, find the drive2 in the GUI and select "eject",
# which should sync and unmount all partitions. Don't be CD'ed
# to drive2 when doing this.
( eject via GUI )
# After the eject, you may see LED activity on drive2 until it stops.
# That is the amount of data you would have lost by just pulling the cable.

It's because of System Write caches, that you have to be careful about
pulling drives.

OK, the system write cache "books" the RAM it uses, unlike
the system read cache. The RAM used by the system write cache
is not returned to the system, until the file is transferred
out to disk.

You can have a "collision" between a system write cache growing, and
some other malloc activity on a machine. The OS design, may take
some of the obvious cases into account -- perhaps it triggers
the OOM killer. But it's also possible to have activity on a
machine, where the OS isn't ready for it. I have frozen an OS
doing this, tight as a drum. Only RESET returned control to me.
That's how dangerous System Write caches are, and why I am not
a fan of them. I can't see how an end-user profits from their
existence. The equipment does not have the observability to be
using them. For example, only the yellow LED on an OCZ Rally
has semantics suited to System Write caches. Most other USB sticks
are hobbled by their "breathing" behavior on the LED, making it
worthless for sensitive applications such as cache flushing.

This is just a wild guess on my part, but if Nemo is
slowing down, in mid transfer, it's possible the System
Write cache is causing swapping and now the swapfile is
engaged as well as writes for the output file.

Paul

On 2024-01-25, Paul wrote:
> [...]
> This is just a wild guess on my part, but if Nemo is
> slowing down, in mid transfer, it's possible the System
> Write cache is causing swapping and now the swapfile is
> engaged as well as writes for the output file.

Write cache won't cause swapping. Things get messy when you talk about
the cache(s); since they're used to make it /appear/ transfers happen
faster. But as you say, you have to wait for a sync() to complete to
flush those to disk.

And this is only looking at the write side -- if there was enough RAM
available (which for an 80G file is slightly improbable on a regular PC,
but not anywhere near impossible), the second transfer with FreeFileSync
could've cut the source drive out completely.

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

On Wed, 24 Jan 2024 13:14:18 -0500, Nic wrote:

> Has any body an explanation why when I copy 40 gb of video files to a
> usb hard drive that is plugged in to a powered usb hub takes over an
> hour to copy while using a software called FreeFileSync only takes 5
> minutes.
>
> I am using NEMO as my file manager to copy from one drive to another. I
> suspect that my file manager may not be a good way to copy large amounts
> of files.

Just for shits and giggles I plugged in an external 240GB Seagate HDD into
a USB 3.1 port on my pc and copied 125GB of data to it using Nemo, elapsed
time 14 mins 35 secs. I did it again using Double Commander, elapsed time
14 mins 10 secs. OP issue might be the USB hub, not saying it is the
problem.

On 2024-01-25, Bubba the Corn Dog wrote:
> On Wed, 24 Jan 2024 13:14:18 -0500, Nic wrote:
>
>> Has any body an explanation why when I copy 40 gb of video files to a
>> usb hard drive that is plugged in to a powered usb hub takes over an
>> hour to copy while using a software called FreeFileSync only takes 5
>> minutes.
>>
>> I am using NEMO as my file manager to copy from one drive to another. I
>> suspect that my file manager may not be a good way to copy large amounts
>> of files.
>
> Just for shits and giggles I plugged in an external 240GB Seagate HDD
> into a USB 3.1 port on my pc and copied 125GB of data to it using
> Nemo, elapsed time 14 mins 35 secs. I did it again using Double
> Commander, elapsed time 14 mins 10 secs. OP issue might be the USB
> hub, not saying it is the problem.

Sounds about on par for the system scheduling out the writes. Without
buffering (i.e. if you can force the CPU to just talk to the USB
device), a USB3 transfer would take approximately 4 minutes (125,000
MB / 500 MB/sec USB3 = 250 sec). USB 3.1 would halve the time ... but
all the datasheets I'm coming across are to the effect of "Up to
SPEED[*]" ([*] SPEED based on sequential read of data.)

Write buffers get messy, because the application can go one of two ways:

(a) "copy" time is only as slow as RD_SRC -> BUF
(b) "copy" time is extended by (lazy) BUF -> WR_TGT

In the case of (a); the copy time will be very short, but then when
trying to eject(1) or umount(8), you'll have to wait your few minutes
for the buffer to be flushed.

In the case of (b); the copy time depends somewhat on the application in
question. It may be the case that the application internally requests a
sync(1) every few GB, in which case, copy times may not be extended all
that much. On the other hand, it may also be the case that the
application simply waits around for the kernel to allow it to write to
its few GB slice of ram-buffer (so the writes would be noticeably slowed
down).

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

On 1/25/24 9:48 AM, Bubba the Corn Dog wrote:
> On Wed, 24 Jan 2024 13:14:18 -0500, Nic wrote:
>
>> Has any body an explanation why when I copy 40 gb of video files to a
>> usb hard drive that is plugged in to a powered usb hub takes over an
>> hour to copy while using a software called FreeFileSync only takes 5
>> minutes.
>>
>> I am using NEMO as my file manager to copy from one drive to another. I
>> suspect that my file manager may not be a good way to copy large amounts
>> of files.
> Just for shits and giggles I plugged in an external 240GB Seagate HDD into
> a USB 3.1 port on my pc and copied 125GB of data to it using Nemo, elapsed
> time 14 mins 35 secs. I did it again using Double Commander, elapsed time
> 14 mins 10 secs. OP issue might be the USB hub, not saying it is the
> problem.
It could be, but for the fact that the same hardware when used with
FreeFileSync showed a great improvement in speed.

On Wed, 24 Jan 2024 17:14:39 -0500
Nic <Nic@none.invalid> wrote:

> On 1/24/24 3:53 PM, Gordon wrote:
> > On 2024-01-24, Big Al <alan@invalid.com> wrote:
> >> On 1/24/24 02:00 PM, Nic wrote:
> >>> On 1/24/24 1:52 PM, Arti F. Idiot wrote:
> >>>> On 1/24/24 11:14 AM, Nic wrote:
> >>>>> Has any body an explanation why when I copy 40 gb of video files to a
> >>>>> usb hard drive that is plugged in to a powered usb hub takes over an
> >>>>> hour to copy while using a software called FreeFileSync only takes 5
> >>>>> minutes.
> >>>>>
> >>>>> I am using NEMO as my file manager to copy from one drive to another. I
> >>>>> suspect that my file manager may not be a good way to copy large
> >>>>> amounts of files.
> >>>>>
> >>>> Nemo is just doing straight copies while "Free"FileSync is doing diffs
> >>>> with a previously populated target drive.  I'd think the *initial*
> >>>> population of the target drive would take about the same amount of
> >>>> time since *all* files are being copied, not just the diffs.
> >>>>
> >>>> Regarding USB mounted memory storage my experience is there can be a
> >>>> great deal of variation is transfer speeds between different brands.
> >>>> And of course having USB 3.1 or 3.2 helps.
> >>>>
> >>>> You could probably match or beat the performance of "Free"FileSync by
> >>>> using rsync or similar archiving scheme.
> >>> I am simply copying from one drive to another. It seems that
> >>> FreeFileSync is using a different method to copy the files than NEMO is
> >>> using.
> >> And that's the same thing I said. Arti and I are trying to say that
> >> FreeFileSync is intelligent. Nemo is dumb.
> > Correct. Another way of looking at it is, that copy and paste is not not the
> > correct tool to use. Copy and paste is really for small files.
> >
> > Rsync's function is to copy (and move) files, big small and lots of them.
> >
> > Grsync is another front end for rsync.
> Thank you. Installed the front end and is working nicely. Pity about
> Nemo, is there a smarter filemanager?

Forgive me for butting in. I have been using rsync for mny years but
still sweat over the switches from time to time.

I tried grsync today from a local to a remote machine on the same LAN.

I have the following questions:
1) Where do the mount points come in?
2) How does one introduce exclusions?

I have "run as superviser" checked.
I keep getting permissions denied.

These are the addresses I have entered:

Source: /home/alan
Target: alan@192.168.1.66:/home/backups

I cannot find anything helpful on the internet. :(

Alan

--
Linux Mint 21.1 kernel version 5.15.0-91-generic Cinnamon 5.6.8
AMD Phenom II x4 955 CPU 16Gb Dram 2TB Barracuda

On 1/24/24 1:53 PM, Gordon wrote:
> ...
> Correct. Another way of looking at it is, that copy and paste is not not the
> correct tool to use. Copy and paste is really for small files.
>
> Rsync's function is to copy (and move) files, big small and lots of them.
>
> Grsync is another front end for rsync.

Grsync has many useful rsync options but it seems not to allow selecting
individual files for copying, just directories. You can manually enter
a filename in though.

One-off file copies with rsync is easy enough in a terminal:

$ rsync -ai <source> <destination>

Making a Nemo script extension shouldn't be too hard.