Hi,

we are frequently cloning windows-systems with the help of NTFS-3G and partclone (also ntfsclone).

Recently we had to update the used linux-system and with that came a switch from NTFS-3G 2014.2.15 to 2016.2.22.

At first everything worked fine with the new version of NTFS-3G - but then we experienced cases were the cloned windows partition was identified as being "non formated" by a Windows based "Hard Disk Manager". Although the partition was working fine under windows and all other hard disc tools showed it as "Ok" we still needed to find what had changed compared to the partitions created with the older version of NTFS-3G.

After some deeper digging with Disc Editors we found that there is a difference in the MFT:
The "Sparse"-Flag is now set to "1" for the second $DATA (0x80)-Attribute of the $BadClus File.

Now the question is if this is an intended change in the behavior for the newer version - meaning is it a feature or a bug?
And more importantly is this setup (having sparse-flag set the $BadClus file) inside of the NTFS specification - meaning can i ignore the one tool that marks the partition as faulty?

Thanks for your help!

Thomas

Hi,


Do you mean the flag is not the same in the original partition and in the cloned one after the cloning, or can this be the consequence of some subsequent action ?

This is probably meaningless as long as no cluster is marked bad (with the $Bad attribute being just a single hole). When a cluster is marked bad, it is recorded in the $Bad attribute, a flags with value 0x8000 is inserted in the "Attribute flags" field of the $Bad attribute, and a flag with value 0x200 is inserted into the general "File attributes".

So, what does the "Runlist" of $Bad attribute look like ? Simply post the output of :


AFAIK ntfs-3g and ntfsclone never make any modification to $BadClus. There are only two situations where $BadClus is modified : by using the --clear-bad-sectors option in ntfsfix, and by resizing the partition by ntfsresize. Did you use one of these tools ? Also did you use chkdsk (which marks bad the clusters it cannot read) ?

There has been a recent change in ntfsresize to discard the bad clusters which are not part of a shrunken partition. This may have caused unintended behavior.

Only Microsoft has access to the NTFS specifications, and we have to guess from what is disclosed or experienced. Experimentally the sparse flags has been found to be set by Windows when at least one of the $DATA attribute is made sparse, and the flag is not cleared when all the holes of the $DATA attribute have been filled.

It should be expected such a tool issues a warning if it finds a cluster marked bad. However there must be some other reason for it to declare the partition as non formatted. Anyway I would be surprised if a wrong sparse flag in $BadClus had any consequence in Windows or ntfs-3g.

Regards

Jean-Pierre

Hi,

I have checked that "ntfsfix --clear-bad-sectors" currently sets the sparse flags. I do not consider this to be wrong, as chkdsk keeps the flags and does not warn about them.

Nevertheless, after formatting the flags are cleared, and they remain so as long as no bad cluster is found. Also, according to your report, third party tools may react unexpectedly.

So I am suggesting the attached patch to clear the flags unconditionally when clearing the bad cluster list. Please test it on your partition on the condition that no bad cluster has been recorded. This will at least tell whether the bad behavior of your tool is caused by the sparse flags, or if something else must be searched for.

Resizing a partition probably also leads to the sparse flags being set. This will be fixed later.

Regards

Jean-Pierre

Hi,

thx for your input.

Exactly - in the original partition the flag was not set.

Here is a picture of the Flag i'm talking about after the cloning process:

I found that if I manually change this to "0" everything is fine and also the "Hard Disk Manager" is ok with the partition.
My "feeling" is that this would be a safe way to fix this issue - but me not being the NTFS expert I wanted to ask your professional opinion.
From your answers i gather that you agree with my assessment? Do you also know a way to apply such a kind of fix under windows in an automated, "end-user friendly" way?

It seems nobody seems to know absolutely if this is Ok or not. I understand that there is no open specification for NTFS and everybody besides Microsoft can only try to get it right. We also tried to get to the bottom of the problem with the "Hard Disk Manager" people and their solution was to fix there software(!!!) but still calling this Attribute "incorrect". I find that solution a bit "unbalanced" or "inconsequent" an tried to get a clearer answer. I could not find a single NTFS-Reference or Analysis which supports there theory but i also could not provide the prove that it is ok to have that sparse flag set. Best would be to demonstrate that this state can also happen during normal operations under windows.
jpa you say:

Does this mean that the sparse flag will be set as soon as the first bad cluster is put in the file? This would then result in the same state - only with the difference that the file would no longer be size 0 but have some actual data? Does that not suggest that this setting is correct or at least allowed?

Again - thx for your feedback and support!

Thomas

Hi again,


Actually, I would like to know :
1) Was the sparse flags set as a consequence of cloning ?
2) Was the sparse flags set as a consequence of something else ? If so, what may have been the cause of the flags being set ?

IMHO the sparse flags are allowed to be set and this is why chkdsk does not complain or clear them.

I can provide ntfsfix for Windows, but most Windows users would not qualify it as "end-user friendly". Also, I would restrict it to administrators.

What suggests that this is correct, or at least allowed, is that chkdsk does not complain or change the flags.

Regards

Jean-Pierre

** edit **

Note : ntfsfix for Windows cannot fix the Windows system partition, it would have to unmount it.