FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Sun Jun 13, 2021 14:03



Post new topic Reply to topic  [ 8 posts ] 
ntfsclone - ignore marked bad clusters 
Author Message

Joined: Fri Oct 30, 2015 02:04
Posts: 5
Post ntfsclone - ignore marked bad clusters
From the experience outlined at http://serverfault.com/questions/732665 ... 811#732811 , I see that `ntfsclone` tries to copy all used clusters, including those marked as bad (since in NTFS, bad sectors are simply assigned to the `$BadClus:$Bad` stream).

Is there a way to make it ignore them and only report read errors outside bad clusters?

If not, consider this a feature request.


Sat Oct 31, 2015 00:05
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsclone - ignore marked bad clusters
Hi,

Quote:
I see that `ntfsclone` tries to copy all used clusters, including those marked as bad

No, ntfsclone only tries to read clusters in use. When chkdsk marks a sector as bad and this sector was in use, it allocates a new sector, so the bad sector is not in use any more.
Quote:
Is there a way to make it ignore them and only report read errors outside bad clusters?

The message "WARNING: The disk has NNNN or more bad sectors..." is a warning about the sectors which were marked bad, and this has no effect on the processing. When ntfsclone fails to read a sector which was not marked bad, it issues an error message and just stops, unless the option --rescue is set.

In the referenced report, another bad sector was apparently found, but when using --rescue, there are several attempts at reading bad (unmarked) sectors, and they were apparently successful.

If you get the warning and you are cloning to a new disk, you should run "ntfsfix --clear-bad-sectors" to unmark the sectors.

Regards

Jean-Pierre


Sat Oct 31, 2015 10:55
Profile

Joined: Fri Oct 30, 2015 02:04
Posts: 5
Post Re: ntfsclone - ignore marked bad clusters
It's hard to argue with the `ntfs-3g`'s lead developer, but I double-checked with `chkdsk` that I have no unmarked bad clusters. (yes, that question was mine, no use hiding this anymore)

I also read https://flatcap.org/linux-ntfs/ntfs/files/badclus.html which says: "all the bad clusters are marked as used in the `$Bitmap` system file, so they can never ever be used by any other file" and checked this by hand:
Code:
Dumping Inode 8 (0x8)
<...>
Dumping attribute $DATA (0x80) from mft record 18 (0x12)
<...>
        Attribute name:          '$Bad'
<...>
        Runlist:        VCN             LCN             Length
                        0x0             <HOLE>          0x4fb0
                        0x4fb0          0x4fb0          0x1
                        0x4fb1          <HOLE>          0x2
<...>
Code:
$ xxd -b <$Bitmap>
<...>
0000097e: 11111111 11111111 11111111 11111111 11111111 11111111  ......
<...>

So, it's kinda strange to hear this from a leading specialist in the domain (unless you're an impostor, that I deem unlikely... for now ;-) ).

I also noticed that the bad cluster list was preserved in the image (so I had to clear it by hand upon restoring to a different drive).

To summarize, the `$BadClus:$Bad` stream appears to receive no special treatment from `ntfsclone` - while `ntfsprogs/ntfsclone.c::is_critical_metadata` shows that it definitely should.
Code:
$ ntfsclone -V
ntfsclone v2015.3.14 (libntfs-3g)
<...>


I'll attach a sample image when I make it and check it - this'll take a while.


Tue Nov 03, 2015 08:55
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsclone - ignore marked bad clusters
Hi,

Quote:
which says: "all the bad clusters are marked as used in the `$Bitmap` system file, so they can never ever be used by any other file"

Did I say otherwise ?
Quote:
So, it's kinda strange to hear this from a leading specialist in the domain (unless you're an impostor, that I deem unlikely... for now ;-) ).

Oh, yes, it is customary for me to say strange things, I must be an imposter.
Quote:
I also noticed that the bad cluster list was preserved in the image (so I had to clear it by hand upon restoring to a different drive).

Of course it does, ntfsclone cannot guess whether you intend to restore back to the same disk or to a new one. And saving the list does not mean saving the bad clusters.
Quote:
To summarize, the `$BadClus:$Bad` stream appears to receive no special treatment from `ntfsclone` - while `ntfsprogs/ntfsclone.c::is_critical_metadata` shows that it definitely should.

But there is the function ignore_bad_clusters() which removes the clusters marked bad from the list of clusters to save. So ntfsclone only saves the clusters which are really in use (and this includes the list of bad clusters).

Regards

Jean-Pierre


Tue Nov 03, 2015 09:51
Profile

Joined: Fri Oct 30, 2015 02:04
Posts: 5
Post Re: ntfsclone - ignore marked bad clusters
Quote:
Did I say otherwise ?

You replied to my "assigned to $Bad" with "No, ntfsclone only tries to read clusters in use. <...> the bad sector is not in use any more." which I interpreted as exactly this. After all, from FS metadata's POV, they are effectively in use. Okay, so that was a misunderstanding.

Quote:
But there is the function ignore_bad_clusters() which removes the clusters marked bad from the list of clusters to save. So ntfsclone only saves the clusters which are really in use


So, you requiire solid evidence that the clusters that `ntfsclone` chokes on are exactly those that are marked as bad. Okay, here goes...


Tue Nov 03, 2015 20:29
Profile

Joined: Fri Oct 30, 2015 02:04
Posts: 5
Post Re: ntfsclone - ignore marked bad clusters
Here:
  • ntfsinfo-badclus.out - `ntfsinfo -i 8 -v <dev>` output
  • ntfsclone-rescue.out - `ntfsinfo --save-image --rescue -o <image> <dev>` output (may have been with `-v`, too)
  • compare-badclus.py - comparison script, should be run as `compare-badclus.py ntfsinfo-badclus.out ntfsclone-rescue.out`
  • compare-badclus.out - script output


Attachments:
File comment: bad clusters vs ntfsclone errors comparison
badclus-compare.7z [16.2 KiB]
Downloaded 863 times
Thu Nov 05, 2015 08:01
Profile

Joined: Fri Oct 30, 2015 02:04
Posts: 5
Post Re: ntfsclone - ignore marked bad clusters
The disk image with user data wiped but bad block list intact:
https://yadi.sk/d/UOHb7binkEr68
(25M, warning - expands to 40G! (can't resize, ntfsresize chokes on it, too :-) )
pipe to the attached script or `cp --sparse` to save space.


Attachments:
File comment: write input as sparse (linux-way)
auto-sparse.py.gz [209 Bytes]
Downloaded 861 times
Thu Nov 05, 2015 09:12
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsclone - ignore marked bad clusters
Hi,

Quote:
The disk image with user data wiped but bad block list intact:

Ok, I see, there were too many bad clusters and they did not fit in a single block...

Patch attached.
Quote:
can't resize, ntfsresize chokes on it, too

Same reason, fix also attached.

Regards

Jean-Pierre


Attachments:
bad-clusters-full.patch.tar.gz [1.8 KiB]
Downloaded 889 times
Mon Nov 09, 2015 09:37
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.