FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Wed May 12, 2021 00:11



Post new topic Reply to topic  [ 14 posts ] 
NTFS-3G recover 
Author Message

Joined: Mon Jan 06, 2014 05:48
Posts: 8
Post NTFS-3G recover
Hi All,
Very thinks for your helps in ntfs-3g tool.
I am using ntfs-3g tool to mount ntfs usb drive in my embedded linux system.
Everything is fine and I could mount it and read/write successfully.
But I met a stranger problem:

1) Formatting the usb drive to ntfs filesystem in my laptop(win8).
2) Uplug it directly without safely umounted.
3) Plug the usb drive to my embedded linux board, ntfs-3g will start to recover the usb drive and mount successfully. Read/write also works fine!
4) Unfortunely, when I plug the usb drive to my laptop again, win8 could not acess this usb drive until formatting again.

My question is:
Does "recover" compatible with win8?
I will try another usb drive and another OS like as win7.

Thanks!
BR
Steven


Mon Jan 06, 2014 06:35
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G recover
Hi,

Quote:
2) Uplug it directly without safely umounted.

I assume you did that on purpose, for testing the consequences of unsafe mounting...
Quote:
3) Plug the usb drive to my embedded linux board, ntfs-3g will start to recover the usb drive

Actually the journal is simply wiped out. This is to prevent the journal to be applied at next mounting on Windows to data which may have been changed in the meantime.
Quote:
Does "recover" compatible with win8?

So far, nobody has been able to understand how the journal is organized, so there is no real recovery in ntfs-3g, just wiping the journal. Nevertheless, as far as I can tell, there is no change in the format : a device unplugged unsafely from Win8 can be properly recovered by Win7. There is however a known change for fixed disks only, in order to signal Win8 was only partially shut down, which is irrelevant in your situation.

Some metadata not used by ntfs-3g was probably not flushed to the device, or it was only marked in the journal.
Quote:
I will try another usb drive and another OS like as win7.

A useful test is to unsafely unplug from Win8, plug into Win7 and safe unplug, then plug into Win8. This would show whether all the data was flushed to the device.

I do not have Win8 any more, can you make an image of the device in the configuration where it can be used by ntfs-3g, but cannot be mounted on Win8 ? Delete your own files first to keep the size to a minimum (use a usb key for testing).
Code:
# as root, with the device unmounted
ntfsclone -s -O - /dev/xxx | bzip2 > imagefile.bz2


Regards

Jean-Pierre


Mon Jan 06, 2014 10:19
Profile

Joined: Mon Jan 06, 2014 05:48
Posts: 8
Post Re: NTFS-3G recover
Hi, Pierre,
Very thanks for your quick reply and sorry for late reply.
I have tried unsafely plug in win7 and win8. The result is the same:
After USB drive formated to NTFS in win7/win8 and unplug unsafely.
Then plug into linux system and mounted by ntfs-3g.
Unplug from linux and plug to win7/win8 again.
Error message will come out:
Quote:

D;\ is not accessible.
The file or directory is corrupted and unreadable.


It's happened randomly.

Quote:
A useful test is to unsafely unplug from Win8, plug into Win7 and safe unplug, then plug into Win8. This would show whether all the data was flushed to the device.

I tried to unsafely unplug between win7 and win8. USB drive is acccessible and all the data seems fine.

Quote:
I do not have Win8 any more, can you make an image of the device in the configuration where it can be used by ntfs-3g, but cannot be mounted on Win8 ? Delete your own files first to keep the size to a minimum (use a usb key for testing).

I have used CHKDSK tool to check unaccessible USB drive. It shows:
Quote:
The type of the file system is NTFS.

CHKDSK is verifying files <stage 1 of 5>...
0 percent complete. <0 of16 file records processed>

CHKDSK seems do nothing...

the unaccessible USB drive could be mounted by ntfs-3g and read/write sucessfully.
I create a test data in the USB drive
Code:
dd if=/dev/zero of=/media/USB2/test bs=1k count=1

It's ok for ntfs-3g mounted but Windows cant acessible also.

The following is ntfsclone command, I cant make an image :
Code:
# ntfsclone -s -o - /dev/sdb1
ntfsclone v2013.1.13 (libntfs-3g)
ERROR: Volume '/dev/sdb1' is scheduled for a check or it was shutdown
uncleanly. Please boot Windows or use the --force option to progress.


Thanks for your support.
Very thanks!
Best regards,
Steven


Fri Jan 10, 2014 07:43
Profile

Joined: Mon Jan 06, 2014 05:48
Posts: 8
Post Re: NTFS-3G recover
steven.weng wrote:
Hi, Pierre,
Very thanks for your quick reply and sorry for late reply.
I have tried unsafely plug in win7 and win8. The result is the same:
After USB drive formated to NTFS in win7/win8 and unplug unsafely.
Then plug into linux system and mounted by ntfs-3g.
Unplug from linux and plug to win7/win8 again.
Error message will come out:
Quote:

D;\ is not accessible.
The file or directory is corrupted and unreadable.


It's happened randomly.

Quote:
A useful test is to unsafely unplug from Win8, plug into Win7 and safe unplug, then plug into Win8. This would show whether all the data was flushed to the device.

I tried to unsafely unplug between win7 and win8. USB drive is acccessible and all the data seems fine.

Quote:
I do not have Win8 any more, can you make an image of the device in the configuration where it can be used by ntfs-3g, but cannot be mounted on Win8 ? Delete your own files first to keep the size to a minimum (use a usb key for testing).

I have used CHKDSK tool to check unaccessible USB drive. It shows:
Quote:
The type of the file system is NTFS.

CHKDSK is verifying files <stage 1 of 5>...
0 percent complete. <0 of16 file records processed>

CHKDSK seems do nothing...

the unaccessible USB drive could be mounted by ntfs-3g and read/write sucessfully.
I create a test data in the USB drive
Code:
dd if=/dev/zero of=/media/USB2/test bs=1k count=1

It's ok for ntfs-3g mounted but Windows cant acessible also.

The following is ntfsclone command, I cant make an image :
Code:
# ntfsclone -s -o - /dev/sdb1
ntfsclone v2013.1.13 (libntfs-3g)
ERROR: Volume '/dev/sdb1' is scheduled for a check or it was shutdown
uncleanly. Please boot Windows or use the --force option to progress.


with -f option :
Code:
# ntfsclone -f -s -o - /dev/sdb1
ntfsclone v2013.1.13 (libntfs-3g)
NTFS volume version: 1.2
Cluster size       : 4096 bytes
Current volume size: 32223784960 bytes (32224 MB)
Current device size: 32223789056 bytes (32224 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Cluster accounting failed at 769805 (0xbbf0d): missing cluster in $Bitmap
Cluster accounting failed at 769806 (0xbbf0e): missing cluster in $Bitmap
Cluster accounting failed at 769807 (0xbbf0f): missing cluster in $Bitmap
Cluster accounting failed at 769808 (0xbbf10): missing cluster in $Bitmap
Cluster accounting failed at 769809 (0xbbf11): missing cluster in $Bitmap
Cluster accounting failed at 769810 (0xbbf12): missing cluster in $Bitmap
Cluster accounting failed at 769811 (0xbbf13): missing cluster in $Bitmap
Cluster accounting failed at 769812 (0xbbf14): missing cluster in $Bitmap
Cluster accounting failed at 769813 (0xbbf15): missing cluster in $Bitmap
Cluster accounting failed at 769814 (0xbbf16): missing cluster in $Bitmap
Totally 65 cluster accounting mismatches.
ERROR: Filesystem check failed! Windows wasn't shutdown properly or inconsistent
filesystem. Please run chkdsk /f on Windows then reboot it TWICE.


Thanks for your support.
Very thanks!
Best regards,
Steven


Fri Jan 10, 2014 07:56
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G recover
Hi,

Code:
Cluster accounting failed at 769814 (0xbbf16): missing cluster in $Bitmap

This means that some cluster was referenced in a file, but not marked as allocated. In such situations, chkdsk normally detects and fixes the bad condition, so there must be something else.
Code:
# ntfsclone -f -s -o - /dev/sdb1
ntfsclone v2013.1.13 (libntfs-3g)
NTFS volume version: 1.2

Oh ! this is an NTFS format for original Windows NT 3 ! Current volume format (since Windows 2000) is 3.1. Did you use some special formatting parameter ? (or maybe formatting goes through several steps and by unplugging you did not let them all run). Older NTFS formats did not have journalling, I am not sure about ntfs-3g behavior with them.

Can you retry doing an image with extra options --ignore-fs-check (and --rescue if needed), but you need the latest ntfsclone for that (2013.1.13AR.3 from http://www.tuxera.com/community/ntfs-3g-advanced/) ?

Regards

Jean-Pierre


Fri Jan 10, 2014 10:36
Profile

Joined: Mon Jan 06, 2014 05:48
Posts: 8
Post Re: NTFS-3G recover
Hi Pierre,
The following is my test cases:
1) USB drive is formatted to NTFS in Windows7 or Windows8.
2) Unplug USB drive unsafely and plug to linux system, then mounted by ntfs-3g
3) Create a test file by the following command(USB drive is mounted in /media/USB2):
Code:
dd if=/dev/zero of=/media/USB2/test bs=1k count=1

4) Make a ntfs image:
Code:
ntfsclone --ignore-fs-check --rescue -f -s -O - /dev/sda1 | bzip2 > imagefile.
bz2

Quote:
ntfsclone v2013.1.13AR.3 (libntfs-3g)
NTFS volume version: 1.2
Cluster size : 4096 bytes
Current volume size: 32223784960 bytes (32224 MB)
Current device size: 32223789056 bytes (32224 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Cluster accounting failed at 769805 (0xbbf0d): missing cluster in $Bitmap
Cluster accounting failed at 769806 (0xbbf0e): missing cluster in $Bitmap
Cluster accounting failed at 769807 (0xbbf0f): missing cluster in $Bitmap
Cluster accounting failed at 769808 (0xbbf10): missing cluster in $Bitmap
Cluster accounting failed at 769809 (0xbbf11): missing cluster in $Bitmap
Cluster accounting failed at 769810 (0xbbf12): missing cluster in $Bitmap
Cluster accounting failed at 769811 (0xbbf13): missing cluster in $Bitmap
Cluster accounting failed at 769812 (0xbbf14): missing cluster in $Bitmap
Cluster accounting failed at 769813 (0xbbf15): missing cluster in $Bitmap
Cluster accounting failed at 769814 (0xbbf16): missing cluster in $Bitmap
Totally 65 cluster accounting mismatches.
WARNING: The NTFS inconsistency was overruled by the --ignore-fs-check option.
Space in use : 69 MB (0.2%)
Saving NTFS to image ...
100.00 percent completed
Syncing ...

5) Unplug and plug into Windows7/8 again, it will show "D:\ is not acessible"

Quote:
Oh ! this is an NTFS format for original Windows NT 3 ! Current volume format (since Windows 2000) is 3.1. Did you use some special formatting parameter ? (or maybe formatting goes through several steps and by unplugging you did not let them all run). Older NTFS formats did not have journalling, I am not sure about ntfs-3g behavior with them.

There are no specail parameters in formatting process. I am using Windows default parameters. After some tests, I found NTFS volume version is different in unplug safely and unsafely...
Unplug safely : NTFS volume version: 3.1
Unplug unsafely: NTFS volume version: 1.2

Quote:
Can you retry doing an image with extra options --ignore-fs-check (and --rescue if needed), but you need the latest ntfsclone for that (2013.1.13AR.3 from http://www.tuxera.com/community/ntfs-3g-advanced/) ?

The attached file is an image that USB drive could be mounted by ntfs-3g but it's not acessible in Windows.

Very thanks for your support!
thanks very much.

Best regards,
Steven


Attachments:
File comment: An NTFS image
imagefile.bz2 [42.92 KiB]
Downloaded 950 times
Mon Jan 13, 2014 07:32
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G recover
Hi,

First, chkdsk for Windows 7 find errors on this file system (so do chkdsk for Windows XP and chkdsk for Vista), but chkdsk for Windows 8 (my own version is a prerelease one) apparently hangs before terminating, so it shows no errors.

Code:
The type of the file system is NTFS.

CHKDSK is verifying files (stage 1 of 3)...
  70 file records processed.                                         
File verification completed.
  0 large file records processed.                                   
  0 bad file records processed.                                     
  0 EA records processed.                                           
Correcting file name errors in system file record segment 9.
  0 reparse records processed.                                     
CHKDSK is verifying indexes (stage 2 of 3)...
70 percent complete. (74 of 84 index entries processed)   
Deleting index entry $ObjId in index $I30 of file 11.
71 percent complete. (75 of 84 index entries processed)   
Deleting index entry $Quota in index $I30 of file 11.
72 percent complete. (76 of 84 index entries processed)   
Deleting index entry $Reparse in index $I30 of file 11.
73 percent complete. (77 of 84 index entries processed)   
Deleting index entry $RmMetadata in index $I30 of file 11.
  84 index entries processed.                                       
Index verification completed.
CHKDSK is scanning unindexed files for reconnect to their original directory.
  1 unindexed files scanned.                                       
Recovering orphaned file $Extend (11) into directory file 5.
  0 unindexed files recovered.                                     
CHKDSK is verifying security descriptors (stage 3 of 3)...
81 percent complete. (7 of 70 file SDs/SIDs processed)   
Replacing missing or invalid security descriptor for file 8.
Replacing missing or invalid security descriptor for file 9.
82 percent complete. (10 of 70 file SDs/SIDs processed)   
Replacing missing or invalid security descriptor for file 11.
  70 file SDs/SIDs processed.                                       
Security descriptor verification completed.
Inserting data attribute into file 9.
  7 data files processed.                                           
Correcting errors in the master file table's (MFT) BITMAP attribute.
Correcting errors in the Volume Bitmap.
Windows has made corrections to the file system.

  31468543 KB total disk space.
         4 KB in 2 files.
         4 KB in 8 indexes.
         0 KB in bad sectors.
     67027 KB in use by the system.
     65536 KB occupied by the log file.
  31401508 KB available on disk.

      4096 bytes in each allocation unit.
   7867135 total allocation units on disk.
   7850377 allocation units available on disk.

Actually chkdsk deletes the metadata files which were not defined for NTFS 1.2, thus retaining compatibility with Windows NT 3.51 (back from 1995).

The errors you got when making the image are related to the metadata file $Secure which did not exist in NTFS 1.2, but were filled with standard security descriptors whose allocation was not marked in the allocation map. This was fixed by chkdsk by keeping the file but clearing its contents ("Inserting data attribute into file 9").

ntfs-3g also maintains the compatibility with NTFS 1.2, and I could check that the security descriptors created by ntfs-3g were recorded the old way. By using secaudit with options -arvv you can check that no security id is defined (this is the old way) for files created by ntfs-3g such as the trash directory /.Trash-1000, but a security id is defined (this is the current way) for three metadata files which were created at formatting time : $UpCase, $Quota and $BadClus.

Code:
secaudit 1.4.1 : NTFS security data auditing

Auditing $SDS-1
** There is no $SDS-1 in this volume

Auditing $SDS-2
** There is no $SDS-2 in this volume

Auditing $SII
** There is no $SII in this volume
       
Auditing $SDH
** There is no $SDH in this volume
[...]
Directory /.Trash-1000
Security key : none
[...]
File /$UpCase
Security key : 0x100
[...]
File /$Quota
Security key : 0x101
[...]
Summary of security key use :
Key 0x100 used by 2 files
Key 0x101 used by 1 file
No errors were found

IMHO ntfs-3g is not to blame in this situation, it has righfully ignored files which were not defined for NTFS 1.2 (though secaudit should have complained on finding security keys which should not have been there). The file system is inconsistent in using a few features defined for NTFS 3.x though claiming to be defined as NTFS 1.2. This apparently disturbs Windows, but ntfs-3g rightfully ignores the files which need not be present.

Also chkdsk for Windows 8 aborting in this situation may mean that Microsoft is considering not supporting NTFS 1.2 any more.

Regards

Jean-Pierre


Mon Jan 13, 2014 11:29
Profile

Joined: Mon Jan 06, 2014 05:48
Posts: 8
Post Re: NTFS-3G recover
Hi,
Thanks for the detailed reply. I am really appreciated for your help.
I am new to NTFS filesystem and still have some questions.
1) Why "ntfsclone" determine the USB drive that unsafely unplug as NTFS v1.2 ?
2) Why Windows cant access the USB drive randomly after ntfs-3g recovering? (Plug and unplug unsafely between Windows is OK.)
Thanks for your patience.
Regards,
Steven


Mon Jan 13, 2014 14:45
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G recover
Hi,
Quote:
1) Why "ntfsclone" determine the USB drive that unsafely unplug as NTFS v1.2 ?

1a) when mounting, Windows marks the device as dirty, and when unmounting it deletes the flag. So when the dirty flag is set, ntfsclone knows there were no unmounting (because of system hang or unsafe unplugging).
1b) ntfsclone determines it is formatted as NTFS v1.2 because the label of the volume says so. probably Windows formats the device in several steps, first as NTFS 1.2, then upgrade to NTFS 3.1, but when you unplugged the device, the upgrade to NTFS 3.1 was not yet synced to final destination (though this must have been written to the journal, waiting for the next sync).
Quote:
2) Why Windows cant access the USB drive randomly after ntfs-3g recovering?

2a) As I mentioned earlier there is no ntfs-3g journal recovery, there is only a journal deletion. Data written to the journal awaiting to be synced is just lost. This can appear as random as it depends on when you unplug the device during the formatting process.
2b) The reasons why Windows 8 cannot access an erroneous NTFS 1.2 file system, after automatically repairing it by chkdsk, are beyond my knowledge.

Regards

Jean-Pierre


Mon Jan 13, 2014 17:09
Profile

Joined: Mon Jan 06, 2014 05:48
Posts: 8
Post Re: NTFS-3G recover
Hi,
Quote:
2a) As I mentioned earlier there is no ntfs-3g journal recovery, there is only a journal deletion. Data written to the journal awaiting to be synced is just lost. This can appear as random as it depends on when you unplug the device during the formatting process.

I trace the ntfs-3g souce code. I found the ntfs_logfile_reset() function in the ntfs_device_mount() of /libntfs-3g/volume.c.
This function mark the journal as empty. Is it that you said there is no ntfs-3g journal recovery, there is only a journal deletion?

I have tried to mark this function(ntfs_logfile_reset()) and rebuild again, then mount as usual in the linux system. It seems ok and Windows could access the USB drive properly after ntfs-3g mounting(I have tried many times).I am doubt that "journal deletion" maybe make the USB drive unaccessible in Windows.

Best regards,
Steven


Tue Jan 14, 2014 12:07
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G recover
Hi,

Quote:
This function mark the journal as empty. Is it that you said there is no ntfs-3g journal recovery, there is only a journal deletion?

Yes.
Quote:
I have tried to mark this function(ntfs_logfile_reset()) and rebuild again, then mount as usual in the linux system. It seems ok and Windows could access the USB drive properly after ntfs-3g mounting(I have tried many times).I am doubt that "journal deletion" maybe make the USB drive unaccessible in Windows

Ok, so you proved the undeleted journal contained what was needed to restore consistency to the file system, and ability to mount it on Windows.
Of course you were in a lucky condition. The journal only contained information about metadata which ntfs-3g did not update because the file system was identified as NTFS 1.2. In a more usual condition (normal files being written by Windows just before unsafe unplugging), ntfs-3g could modify metadata still in a temporary state, and Windows would apply the journal later with some confusion about the metadata updated by ntfs-3g in the meantime. This is why the journal should be deleted.

Regards

Jean-Pierre


Tue Jan 14, 2014 13:24
Profile

Joined: Mon Jan 06, 2014 05:48
Posts: 8
Post Re: NTFS-3G recover
Hi Pierre,
Thanks a lot.
Quote:
Ok, so you proved the undeleted journal contained what was needed to restore consistency to the file system, and ability to mount it on Windows.

Do you have any suggestions or ideas about this issue?
It seems the best way to get rid of this issue is mounted with norecover option...and always unplug safely on Windows.
Thanks again.
Best regards,
Steven


Tue Jan 14, 2014 16:17
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G recover
Hi,

Quote:
Do you have any suggestions or ideas about this issue?

Yes, an obvious one : process the journal. You can contribute, of course, just be aware a few developers have unsuccessfully tried. What has been identified so far is analyzed in the source code ntfsprogs/ntfsdump_logfile.c and this is far from what is needed.
Quote:
It seems the best way to get rid of this issue is mounted with norecover option...and always unplug safely on Windows.

Sure. Earlier, the option norecover was the default, but users complained about the hassle of having to mount on Windows again when a device was unsafely unplugged. As the journal only contains unsynced though committed metadata updates, with no user data, the current default option is generally acceptable.

* edit *

In the image you posted, the flag VOLUME_UPGRADE_ON_MOUNT is set. This probably means that the formatting process on Windows is minimal and restricted to NTFS 1.2, and the upgrade to the currently supported NTFS version is only done at the first subsequent mounting.

Regards

Jean-Pierre


Tue Jan 14, 2014 22:33
Profile

Joined: Mon Jan 06, 2014 05:48
Posts: 8
Post Re: NTFS-3G recover
Hi Jean-Pierre,
Quote:
In the image you posted, the flag VOLUME_UPGRADE_ON_MOUNT is set. This probably means that the formatting process on Windows is minimal and restricted to NTFS 1.2, and the upgrade to the currently supported NTFS version is only done at the first subsequent mounting.

Yes, In the case of formatting process that completed on Windows and unplug USB drive immediately. The USB drive may restricted to NTFS v1.2.
After a while or write a file in NTFS volume, the NTFS volume version will be upgraded to V3.1.

Very thanks for your help.
Thanks again.

Regards,
Steven


Wed Jan 15, 2014 03:53
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.