FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Tue May 11, 2021 02:45



Post new topic Reply to topic  [ 5 posts ] 
$BadClus File Sparse flag set "1" for $DATA (0x80)-Attribute 
Author Message

Joined: Wed Jun 29, 2016 16:06
Posts: 2
Post $BadClus File Sparse flag set "1" for $DATA (0x80)-Attribute
Hi,

we are frequently cloning windows-systems with the help of NTFS-3G and partclone (also ntfsclone).

Recently we had to update the used linux-system and with that came a switch from NTFS-3G 2014.2.15 to 2016.2.22.

At first everything worked fine with the new version of NTFS-3G - but then we experienced cases were the cloned windows partition was identified as being "non formated" by a Windows based "Hard Disk Manager". Although the partition was working fine under windows and all other hard disc tools showed it as "Ok" we still needed to find what had changed compared to the partitions created with the older version of NTFS-3G.

After some deeper digging with Disc Editors we found that there is a difference in the MFT:
The "Sparse"-Flag is now set to "1" for the second $DATA (0x80)-Attribute of the $BadClus File.

Now the question is if this is an intended change in the behavior for the newer version - meaning is it a feature or a bug?
And more importantly is this setup (having sparse-flag set the $BadClus file) inside of the NTFS specification - meaning can i ignore the one tool that marks the partition as faulty?

Thanks for your help!

Thomas


Thu Jun 30, 2016 14:54
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: $BadClus File Sparse flag set "1" for $DATA (0x80)-Attribute
Hi,

Quote:
After some deeper digging with Disc Editors we found that there is a difference in the MFT:
The "Sparse"-Flag is now set to "1" for the second $DATA (0x80)-Attribute of the $BadClus File.

Do you mean the flag is not the same in the original partition and in the cloned one after the cloning, or can this be the consequence of some subsequent action ?
Quote:
After some deeper digging with Disc Editors we found that there is a difference in the MFT:
The "Sparse"-Flag is now set to "1" for the second $DATA (0x80)-Attribute of the $BadClus File.

This is probably meaningless as long as no cluster is marked bad (with the $Bad attribute being just a single hole). When a cluster is marked bad, it is recorded in the $Bad attribute, a flags with value 0x8000 is inserted in the "Attribute flags" field of the $Bad attribute, and a flag with value 0x200 is inserted into the general "File attributes".

So, what does the "Runlist" of $Bad attribute look like ? Simply post the output of :
Code:
ntfsinfo -fvi 8 /dev/partition

Quote:
Now the question is if this is an intended change in the behavior for the newer version - meaning is it a feature or a bug?

AFAIK ntfs-3g and ntfsclone never make any modification to $BadClus. There are only two situations where $BadClus is modified : by using the --clear-bad-sectors option in ntfsfix, and by resizing the partition by ntfsresize. Did you use one of these tools ? Also did you use chkdsk (which marks bad the clusters it cannot read) ?
Quote:
Now the question is if this is an intended change in the behavior for the newer version - meaning is it a feature or a bug?

There has been a recent change in ntfsresize to discard the bad clusters which are not part of a shrunken partition. This may have caused unintended behavior.
Quote:
And more importantly is this setup (having sparse-flag set the $BadClus file) inside of the NTFS specification

Only Microsoft has access to the NTFS specifications, and we have to guess from what is disclosed or experienced. Experimentally the sparse flags has been found to be set by Windows when at least one of the $DATA attribute is made sparse, and the flag is not cleared when all the holes of the $DATA attribute have been filled.
Quote:
meaning can i ignore the one tool that marks the partition as faulty?

It should be expected such a tool issues a warning if it finds a cluster marked bad. However there must be some other reason for it to declare the partition as non formatted. Anyway I would be surprised if a wrong sparse flag in $BadClus had any consequence in Windows or ntfs-3g.

Regards

Jean-Pierre


Thu Jun 30, 2016 22:24
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: $BadClus File Sparse flag set "1" for $DATA (0x80)-Attribute
Hi,

I have checked that "ntfsfix --clear-bad-sectors" currently sets the sparse flags. I do not consider this to be wrong, as chkdsk keeps the flags and does not warn about them.

Nevertheless, after formatting the flags are cleared, and they remain so as long as no bad cluster is found. Also, according to your report, third party tools may react unexpectedly.

So I am suggesting the attached patch to clear the flags unconditionally when clearing the bad cluster list. Please test it on your partition on the condition that no bad cluster has been recorded. This will at least tell whether the bad behavior of your tool is caused by the sparse flags, or if something else must be searched for.
Code:
ntfsfix --clear-bad-sectors --clear-dirty /dev/partition

Resizing a partition probably also leads to the sparse flags being set. This will be fixed later.

Regards

Jean-Pierre


Attachments:
clear-sparse.patch.gz [1005 Bytes]
Downloaded 600 times
Fri Jul 01, 2016 17:01
Profile

Joined: Wed Jun 29, 2016 16:06
Posts: 2
Post Re: $BadClus File Sparse flag set "1" for $DATA (0x80)-Attribute
Hi,

thx for your input.

Quote:
Do you mean the flag is not the same in the original partition and in the cloned one after the cloning, or can this be the consequence of some subsequent action ?
Exactly - in the original partition the flag was not set.

Here is a picture of the Flag i'm talking about after the cloning process:
Attachment:
snip.png
snip.png [ 253.07 KiB | Viewed 11826 times ]

I found that if I manually change this to "0" everything is fine and also the "Hard Disk Manager" is ok with the partition.
My "feeling" is that this would be a safe way to fix this issue - but me not being the NTFS expert I wanted to ask your professional opinion.
Quote:
I have checked that "ntfsfix --clear-bad-sectors" currently sets the sparse flags. I do not consider this to be wrong, as chkdsk keeps the flags and does not warn about them.
From your answers i gather that you agree with my assessment? Do you also know a way to apply such a kind of fix under windows in an automated, "end-user friendly" way?

It seems nobody seems to know absolutely if this is Ok or not. I understand that there is no open specification for NTFS and everybody besides Microsoft can only try to get it right. We also tried to get to the bottom of the problem with the "Hard Disk Manager" people and their solution was to fix there software(!!!) but still calling this Attribute "incorrect". I find that solution a bit "unbalanced" or "inconsequent" an tried to get a clearer answer. I could not find a single NTFS-Reference or Analysis which supports there theory but i also could not provide the prove that it is ok to have that sparse flag set. Best would be to demonstrate that this state can also happen during normal operations under windows.
jpa you say:
Quote:
Nevertheless, after formatting the flags are cleared, and they remain so as long as no bad cluster is found.

Does this mean that the sparse flag will be set as soon as the first bad cluster is put in the file? This would then result in the same state - only with the difference that the file would no longer be size 0 but have some actual data? Does that not suggest that this setting is correct or at least allowed?

Again - thx for your feedback and support!

Thomas


Fri Jul 01, 2016 17:46
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: $BadClus File Sparse flag set "1" for $DATA (0x80)-Attribute
Hi again,

Quote:
Quote:
Do you mean the flag is not the same in the original partition and in the cloned one after the cloning, or can this be the consequence of some subsequent action ?

Exactly - in the original partition the flag was not set.

Actually, I would like to know :
1) Was the sparse flags set as a consequence of cloning ?
2) Was the sparse flags set as a consequence of something else ? If so, what may have been the cause of the flags being set ?
Quote:
From your answers i gather that you agree with my assessment?

IMHO the sparse flags are allowed to be set and this is why chkdsk does not complain or clear them.
Quote:
Do you also know a way to apply such a kind of fix under windows in an automated, "end-user friendly" way?

I can provide ntfsfix for Windows, but most Windows users would not qualify it as "end-user friendly". Also, I would restrict it to administrators.
Quote:
Does that not suggest that this setting is correct or at least allowed?

What suggests that this is correct, or at least allowed, is that chkdsk does not complain or change the flags.

Regards

Jean-Pierre

** edit **

Note : ntfsfix for Windows cannot fix the Windows system partition, it would have to unmount it.


Fri Jul 01, 2016 19:54
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.