FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Mon May 10, 2021 10:36



Post new topic Reply to topic  [ 4 posts ] 
System gets inoperable while copying large amounts of files 
Author Message

Joined: Fri Oct 16, 2015 22:30
Posts: 3
Post System gets inoperable while copying large amounts of files
Full subject:
System gets inoperable when copying great amounts of files across drives

(Note: this is a tech forum, so the max length specified by the admin for subjects is definitely set to a too low value.)

I'm on Ubuntu Wily using 2014.2.15AR.3-3, but this problem has been reproducible since Saucy.

I assume it only occurs when copying files from IDE drive to IDE drive, or from IDE to SATA drive and vice versa. Must be a timing error or so.
SATA to SATA (normally) works fine.

I will give some details:

* Supposedly you have 5,000 files to copy. Source and destination being both NTFS, so that you can access these files with both Linux and Windows.

* Both hard drives would show a 100% Health value in HDD Sentinel. No defective sectors.

* You open a simple file manager, say PCManFM, and start copying.

* From one point (let's say 66%), it will be stuck at one and the same file.

* It is unlikely to be related to the kernel. Has happened in both Linux 3 and 4 throughout.

From this moment on, the only way out is to power-cycle the system!!

x You cannot click "Stop" in the "Copying..." dialog box.

x Only a handful of tools will work (ls, lsblk etc.)

x When starting one of the consoles (CTRL-ALT-F{1..4}), you try to log in as 'root' to kill the process, but you cannot. It doesn't get to the Password prompt anymore. System is quasi-unusable.

x Even dmesg and shutdown will freeze and not execute (likewise most of other tools)

x Consequently, copying this amount of files may have to be split up into several chunks, to not trigger the "death point" again. Once you do, it's rebooting time again.
(When copying 45 GB once, I had to reboot over 20 times until I got everything to the other side. What a nightmare.)

x Keeping an image of the target drive is a wise thing to do. Because a clean shutdown is never possible at this state, you're likely to DESTROY your whole file structure on the target drive because you're forced to power-cycle in mid-air to get your task done.

On an older system, I once was able to get a dmesg output, and it spit out some message from the kernel like "Process suspended for more than 120 seconds" (IIRC).


Fri Oct 16, 2015 22:52
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: System gets inoperable while copying large amounts of files
Hi,

Quote:
From one point (let's say 66%), it will be stuck at one and the same file.

Anything special about this file (size, compression, etc.) ?
Quote:
x You cannot click "Stop" in the "Copying..." dialog box.

x Only a handful of tools will work (ls, lsblk etc.)

x When starting one of the consoles (CTRL-ALT-F{1..4}), you try to log in as 'root' to kill the process, but you cannot. It doesn't get to the Password prompt anymore. System is quasi-unusable.

x Even dmesg and shutdown will freeze and not execute (likewise most of other tools)

x Consequently, copying this amount of files may have to be split up into several chunks, to not trigger the "death point" again. Once you do, it's rebooting time again.

This looks like the system went out of memory and began swapping, thus putting even more strain on disks. Most likely your file manager triggered too many copies in parallel. Did you try using basic "cp" (and option -r) for copying your files ?

To asses which process exhausted the available memory, start "top" in a terminal windows before starting the copy and keep "top" in a visible windows. When the system freezes make a picture of the windows with a digital camera.

Regards

Jean-Pierre


Sat Oct 17, 2015 08:37
Profile

Joined: Fri Oct 16, 2015 22:30
Posts: 3
Post Re: System gets inoperable while copying large amounts of files
Hello, and thanks for the quick reply.

jpa wrote:
Hi,
This looks like the system went out of memory and began swapping, thus putting even more strain on disks. Most likely your file manager triggered too many copies in parallel. Did you try using basic "cp" (and option -r) for copying your files ?


Yes, the latter usually works.
And there is the next problem: all the file managers I checked through depend on the SAME library (i. e. libfm), so there is hardly any point changing the program unless it uses something different from libfm.

However, what contradicts your theory is that it DOES work no matter what when copying extX to extY (w/ X = 3 (or 4), Y = 3 ((or 4)).
The problem solely happens when I have to deal with NTFS partitions, and mainly if I copy from NTFS to NTFS.


Sun Oct 18, 2015 21:16
Profile

Joined: Fri Oct 16, 2015 22:30
Posts: 3
Post Re: System gets inoperable while copying large amounts of files
Got the screenshot. (Finally!)

Shortly after it froze (not even making it to 25% when copying!), scrot was still operational for a short time. This is from a session *after reboot*, so there is no other application running at the same time except the DE, the window manager, pcmanfm, network-manager and volume control.

That's when I could make 2 'top' screenshots, the best of which I'll post as follows:


Attachments:
ntfscopy_pcmanfm_tasksusp_1gPhyram_3gSwap.jpg
ntfscopy_pcmanfm_tasksusp_1gPhyram_3gSwap.jpg [ 231.67 KiB | Viewed 15374 times ]
Sun Nov 15, 2015 23:12
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.