FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Fri Jun 18, 2021 20:38



Post new topic Reply to topic  [ 20 posts ] 
ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn 
Author Message

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,
I'm trying to (downward) extend an NTFS volume and keep getting errors/warnings on no-action runs of ntfsresize. I got the same "ntfs_mst_post_read_fixup_warn" before when resizing another volume the same way but deemed it non-critical and went ahead without "-n". Unfortunately the process failed and I stupidly didn't save the error message (some MFT number couldn't be read). Since I was concerned about data integrity (altough it seemed to work fine after I restored the original partition table) I restored everything from a backup.

Now this time I'd like to avoid making the same mistake again so I'd really be interested in the meaning of the possible error message and whether it is safe to continue using the volume with the original smaller size after a downward resize was interrupted.

my current disk layout:
Code:
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00050b37

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   251660287   125829120    7  HPFS/NTFS/exFAT
/dev/sdb2       503318528  1953525167   725103320    7  HPFS/NTFS/exFAT


extended 2nd partition:
Code:
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   251660287   125829120    7  HPFS/NTFS/exFAT
/dev/sdb2       251660288  1953525167   850932440    7  HPFS/NTFS/exFAT


output of ntfsresize -x -n /dev/sdb2:
Code:
ntfsresize v2013.1.13AR.1 (libntfs-3g)
Sectors allocated to volume :  old 1450206632 current 1701864872 difference 251658240
Clusters allocated to volume : old 181275829 current 212733109 difference 31457280

*** WARNING ***

Expanding a volume is an experimental new feature

Should something go wrong during the actual resizing (power outage, etc.),
just restart the procedure, but DO NOT TRY to repair with chkdsk or similar,
until the resizing is over, you would LOSE YOUR DATA !

You have been warned !

100.00 percent completed
ntfs_mst_post_read_fixup_warn: magic: 0x00000000  size: 1024   usa_ofs: 0  usa_count: 65535: Invalid argument
ntfs_mst_post_read_fixup_warn: magic: 0x00000000  size: 1024   usa_ofs: 0  usa_count: 65535: Invalid argument
ntfs_mst_post_read_fixup_warn: magic: 0x00000000  size: 1024   usa_ofs: 0  usa_count: 65535: Invalid argument
ntfs_mst_post_read_fixup_warn: magic: 0x00000000  size: 1024   usa_ofs: 0  usa_count: 65535: Invalid argument

All checks have been completed successfully
Cannot check further in no-action mode


Mon Sep 22, 2014 12:09
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

Quote:
I'm trying to (downward) extend an NTFS volume and keep getting errors/warnings on no-action runs of ntfsresize. I got the same "ntfs_mst_post_read_fixup_warn" before when resizing another volume the same way but deemed it non-critical and went ahead without "-n". Unfortunately the process failed and I stupidly didn't save the error message (some MFT number couldn't be read).

In the context of ntfsresize, the "ntfs_mst_post_read_fixup_warn" is not important. This happens because ntfsresize has to recompute all MFT entries and among them there are entries which have never been used, so they contain uninitialized data with wrong integrity checks. So just ignore the warnings unless they lead to real errors (in which case, ntfsresize refuses to proceed).
Quote:
whether it is safe to continue using the volume with the original smaller size after a downward resize was interrupted.

When expanding downward, new metadata is built within the added space with no change at all to the original data, so you only have to rebuild the original partition layout to get your data back. Also, if you interrupt the resizing for some reason, you can restart it over again.

Regards

Jean-Pierre


Mon Sep 22, 2014 19:15
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Thanks for getting back so quickly! So I went ahead and gave ntfsresize -x a go but it immediately terminated with this error:
Code:
ERROR: Could not read the $MFT entry 40

I'm a bit unclear about terminology here but I assume that's what you can have ntfsinfo look up at inode 40, so:
Code:
# ntfsinfo -i40 /dev/sdb2
Error loading node: No such file or directory

Would that be due to a corrupt/unreadable file somewhere? Is there any way to identify which one?

Windows 7 chkdsk keeps finding no problem with the filesystem.
I'm attaching the output of "ntfsinfo -m -v /dev/sdb2", maybe it'll help somehow.


Attachments:
File comment: ntfsinfo -m -v /dev/sdb2
ntfsinfo_sdb2.gz [581 Bytes]
Downloaded 1071 times
Mon Sep 22, 2014 23:28
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Unfortunately I can't seem to edit my previous post, so sorry for spamming the forum.

It just crossed my mind to have a look at ntfsundelete and sure enough, the first hit seems to be the (former) file in question:
Code:
Inode    Flags  %age  Date           Size  Filename
---------------------------------------------------------------
40       F..!     0%  1970-01-01         0  <none>
90529    F..!     0%  1970-01-01         0  <none>
124402   F..!     0%  1970-01-01         0  <none>
129190   F..!     0%  1970-01-01         0  <none>
162519   D..!     0%  2014-04-01         0  done
217217   F..!     0%  1970-01-01         0  <none>
217218   F..!     0%  1970-01-01         0  <none>
...

There are quite a few more inodes with missing filenames and attributes (I just grepped for '!').
Are they what's causing ntfsresize to fail?


Mon Sep 22, 2014 23:50
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

Quote:
Windows 7 chkdsk keeps finding no problem with the filesystem.

Never use chkdsk when the file system does not match the partition, with option /F it will destroy all metadata, (and without /F its output is meaningless).
Quote:
There are quite a few more inodes with missing filenames and attributes (I just grepped for '!').

These MFT entries have no name and no date, they have never been used and they are meaningless. No worry.
Quote:
Are they what's causing ntfsresize to fail?

Apparently, yes, there must be a bug somewhere.

Can you wait about one week for a fix or workaround ? I am short of time now.

Regards

Jean-Pierre


Tue Sep 23, 2014 12:28
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
jpa wrote:
Never use chkdsk when the file system does not match the partition, with option /F it will destroy all metadata, (and without /F its output is meaningless).

Absolutely, I was referring to the original partition/filesystem sizes of course.

jpa wrote:
Can you wait about one week for a fix or workaround ? I am short of time now.

Sure thing!


Tue Sep 23, 2014 13:56
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

Can you post a dump of the MFT entry 40, so that I can be sure to hunt the right issue ? According to your attachment, it was located at cluster 786442 in the original partition, hence at cluster 32243722 in the expanded partition (the MFT started at cluster 786432, and there are 4 entries per cluster, hence entry 40 was at 786432+40/4, then you added 31457280 clusters).

Using the new partition layout, you can :
Code:
dd if=/dev/sdb2 bs=4096 skip=32243722 count=1 | od -t x1 | head -n 16

and post what you get. The beginning should look like this :
Code:
0000000 46 49 4c 45 ....


Regards

Jean-Pierre


Tue Sep 23, 2014 17:45
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hey Jean-Pierre,
this is the data I got:
Code:
0000000 46 49 4c 45 30 00 03 00 00 00 00 00 00 00 00 00
0000020 28 00 00 00 38 00 00 00 40 00 00 00 00 04 00 00
0000040 00 00 00 00 00 00 00 00 04 00 00 00 28 00 00 00
0000060 03 00 00 00 00 00 00 00 ff ff ff ff 00 00 00 00
0000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
0000760 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 00
0001000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
0001760 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 00
0002000 46 49 4c 45 30 00 03 00 00 00 00 00 00 00 00 00
0002020 29 00 01 00 38 00 01 00 e0 01 00 00 00 04 00 00
0002040 00 00 00 00 00 00 00 00 04 00 00 00 29 00 00 00
0002060 01 00 00 00 00 00 00 00 10 00 00 00 48 00 00 00
0002100 00 00 18 00 00 00 00 00 30 00 00 00 18 00 00 00
0002120 79 66 20 fc 23 d7 cf 01 79 66 20 fc 23 d7 cf 01


Tue Sep 23, 2014 17:56
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

I have retried resizing a partition with the same parameters as yours, without getting into any error. There must be something special in your configuration. The inode 40 has clearly never been used, and the "ERROR: Could not read the $MFT entry 40" implies it was allocated, so there is an inconsistency which chkdsk should have caught. Maybe there is a hardware error affecting the bitmap.

Can you retry the resizing in verbose mode, storing the log, so that I can see at which stage the error occurs :
Code:
ntfsresize -v -x /dev/sdb2 | gzip > ntfsresize.log.gz

Note that you will not see the "want to proceed" question, so you have to blindly type a "y".
Then please post the ntfsresize.log.gz as an attachment (compression is mandated by this forum).

I may later have to ask you a full copy of the metadata, but this implies reverting to the initial partition layout. This is how you can do it (but do not post the metadata until I ask).
Code:
ntfsclone -mst -O - /dev/sdb2 | gzip > metadata.gz


Regards

Jean-Pierre


Wed Sep 24, 2014 10:56
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hey Jean-Pierre,

first of all I apologize for probably wasting your time, but the system in question is not technically mine, it's my girlfriend's, and she wanted access to her files. So there was probably data written to the NTFS volume in question (I had restored the partition layout from the original MBR), since the ntfsresize error now occurs at MFT entry 58671 instead of 40 (see log).

Nevertheless I've made a backup of the metadata right before the logged resize operation (taking care of the changing partition sizes of course), so at least those two should refer to the same state on disk this time.

jpa wrote:
Maybe there is a hardware error affecting the bitmap.

S.M.A.R.T. data looks fine, at least there is nothing like write errors or reallocated sectors.


Attachments:
ntfsresize.log.gz [176.14 KiB]
Downloaded 1054 times
Wed Sep 24, 2014 13:26
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

Quote:
first of all I apologize for probably wasting your time,

Not at all. Getting rid of bugs my aim, and this is probably the first report about resizing downward, so I have to be sure there is no hidden misbehavior.
Quote:
So there was probably data written to the NTFS volume in question (I had restored the partition layout from the original MBR)

After changing the partition layout, it is best to reboot. I have seen reports by Gparted related to some obsolete partition information being used somewhere.
Quote:
since the ntfsresize error now occurs at MFT entry 58671 instead of 40 (see log).

Your resizing log only shows very basic actions, the partition has probably been defragmented recently. I am the more surprised to find an error which I can only explain by a bitmap inconsistency and which disappears with no clear reason.

Are you using some hardware or software cacheing on a SSD such as Intel Smart Response, Expresscache or similar ? They are not both Linux and Windows compatible, and this would also explain the change of behavior about inode 40. If this is the case, and if your friend only uses Windows, you need the Windows port of ntfsresize (which I have not published yet because I thought it unuseful).

Regards

Jean-Pierre


Wed Sep 24, 2014 16:42
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
jpa wrote:
After changing the partition layout, it is best to reboot. I have seen reports by Gparted related to some obsolete partition information being used somewhere.

This was after a reboot, under Windows. The partition is pretty much exclusively written to and read from Windows. I don't think I've ever mounted it under Linux.

Or do you mean a reboot is advisable between changing the partition layout (with fdisk, I don't use gparted) and using ntfsresize? I didn't do that.

jpa wrote:
the partition has probably been defragmented recently.

Not really, AFAIK. At least there is no automatic defragmentation scheduled under Windows and I don't remember doing it manually. As far as I am aware, the relevant NTFS volume is just not in very heavy use. It is basically only being used to store jpegs and h264 video from a digital camera. Some files might get deleted, but they should never shrink or grow in size. Although the statistics (wiped deleted) from ntfsclone -mst seem to suggest something different, don't they?
Code:
ntfsclone v2013.1.13AR.1 (libntfs-3g)
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 742505795584 bytes (742506 MB)
Current device size: 742505799680 bytes (742506 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Space in use       : 271 MB (0,0%)   
Scanning volume ...
100.00 percent completed
Num of MFT records       =     217344
Num of used MFT records  =      59099
Wiped unused MFT data    =     358263
Wiped deleted MFT data   =   31924022
Wiped resident user data =    2919022
Wiped timestamp data     =          0
Wiped totally            =   35201307
jpa wrote:
I am the more surprised to find an error which I can only explain by a bitmap inconsistency and which disappears with no clear reason.

I though the reason for MFT entry 40 no longer causing the error is that it's now being used for an actual file, right? I checked and there was definitely a new directory with a few hundred MB of data copied to the volume (with original size restored) after I posted that sector dump. That's why I thought that specific instance of the problem is pretty much worthless now.

I could still upload the ntfsclone image with the MFT data if you think you'll need it. It's from the same time as the ntfsresize log I posted so nothing changed in between. It is about 10MB gzipped.

jpa wrote:
Are you using some hardware or software cacheing on a SSD such as Intel Smart Response, Expresscache or similar ?

None whatsoever. It's a very basic setup with one SSD & one HDD, nothing fancy going on there.


Wed Sep 24, 2014 17:58
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

Quote:
Or do you mean a reboot is advisable between changing the partition layout (with fdisk, I don't use gparted) and using ntfsresize?

Yes, I have seen reports about obsolete partition data still being used in the kernel, but I never have had this problem, and it may have been fixed now.
Quote:
At least there is no automatic defragmentation scheduled under Windows and I don't remember doing it manually.

Well, I was expecting some complex configuration, and I was surprised there were only basic ones...
Quote:
I could still upload the ntfsclone image with the MFT data if you think you'll need it. It's from the same time as the ntfsresize log I posted so nothing changed in between. It is about 10MB gzipped.

I am short of explanation, so I suggest you make the metadata image available next week, I will rebase it myself, fixing ntfsresize issues if any, and sending you back the rebased metadata. You will only have to plug it into the added space to get the resizing done (provided you do not update anything in the meantime). But I cannot do it now.
Quote:
It's a very basic setup with one SSD & one HDD, nothing fancy going on there.

So there is a SSD. Do you know what it is used for ?

Regards

Jean-Pierre


Wed Sep 24, 2014 22:30
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hey Jean-Pierre,
jpa wrote:
Quote:
Or do you mean a reboot is advisable between changing the partition layout (with fdisk, I don't use gparted) and using ntfsresize?

Yes, I have seen reports about obsolete partition data still being used in the kernel, but I never have had this problem, and it may have been fixed now.

Just tried it, didn't change a thing.
jpa wrote:
Well, I was expecting some complex configuration, and I was surprised there were only basic ones...

Yeah, well, there you have it ;)
jpa wrote:
I am short of explanation, so I suggest you make the metadata image available next week, I will rebase it myself, fixing ntfsresize issues if any, and sending you back the rebased metadata.

Thanks, but you don't need to do that. Remember I tried a downward resizing before (namely the other partition on the disk) which failed just the same way. Formatting and restoring from backup took care of that. And I might consider the same for this one if need be and get some nice, fresh, clean MFT in the process ;).

But as a matter of fact I'd be more interested in finding out what's wrong with the filesystems I have. So like I said, there is absolutely no hurry on my side! I won't have access to this PC in the next couple of weeks anyway. Just thought you could have a look at the MFT data when it's convenient and maybe find some sort of fix.

Just one additional, possibly irrelevant obervation adding to the weird behavior: Since ntfsresize didn't like inode 58671 I thought I'd have a look at other inodes close by. Calling ntfsinfo -i on anything (within reason) after 58660 yields the error, even though ntfsresize didn't complain.
Code:
Error loading node: No such file or directory

So I guess the problem might not lie with ntfsresize after all. I don't know, something else may be wrong here. Think I should try a more recent ntfsprogs release or some Linux LiveCD? I'm currently on LMDE.
jpa wrote:
So there is a SSD. Do you know what it is used for ?

Yes, I know since I set it up myself. There is not one sector unaccounted for where some caching mechanism might be hiding. Just two partitions, Win7/Linux and no other (USB, mSATA or whatever) storage connected to the system.


Thu Sep 25, 2014 00:34
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

Quote:
But as a matter of fact I'd be more interested in finding out what's wrong with the filesystems I have. So like I said, there is absolutely no hurry on my side! I won't have access to this PC in the next couple of weeks anyway. Just thought you could have a look at the MFT data when it's convenient and maybe find some sort of fix.

I hope I will be able to explain.

Also, unfortunately the way I suggested is wrong : I forgot small files are stored within the MFT, by plugging the new metadata onto the added space, you would lose those files.
Quote:
Since ntfsresize didn't like inode 58671 I thought I'd have a look at other inodes close by. Calling ntfsinfo -i on anything (within reason) after 58660 yields the error, even though ntfsresize didn't complain.

This is normal behavior : ntfsinfo can only display inodes which are in use. So is ntfsresize expected to do, and ntfsresize should not examine unused ones and fail.
Quote:
Think I should try a more recent ntfsprogs release or some Linux LiveCD? I'm currently on LMDE.

Maybe you try a live CD (different kernel and ntfsprogs versions).

Regards

Jean-Pierre


Thu Sep 25, 2014 06:54
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hey Jean-Pierre,

just finished testing with the latest version v2014.2.15 (sources directly off your website) on my current system with Linux 3.11-2-amd64 (LMDE). Also with version v2012.1.15AR.8 on a 3.16.2 kernel (Knoppix 7.4). All keep running into the exact same problem at inode 58671.


Fri Sep 26, 2014 17:07
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

Sorry for may late reply.

Please upload to some server the metadata image you made, so that I can download it (post its URL as a PM). If you cannot, you will have to split it into three pieces and send it by mail.

Regards

Jean-Pierre


Mon Sep 29, 2014 22:02
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hi,

Yes, there was a bug : when fragmented, the bitmap was wrongly read. Please try the attached patch (if you do not how to apply it, just ask).

You may now delete the uploaded image.

Thank you for reporting the issue and helping to solve it.

Regards

Jean-Pierre


Attachments:
ntfsresize.c.patch.gz [290 Bytes]
Downloaded 1049 times
Tue Sep 30, 2014 09:52
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Alright, that was quick!
I will test your patch as soon as I get home and have access to the system again which unfortunately, like I said before, won't be for a few weeks.
But thanks very much indeed for your effort! I'll get back to you as soon as I can.


Tue Sep 30, 2014 14:58
Profile

Joined: Mon Sep 22, 2014 11:51
Posts: 11
Post Re: ntfsresize fails(?) with ntfs_mst_post_read_fixup_warn
Hey Jean-Pierre,

sorry for the long delay but I finally had a chance to test your patch (against ntfsresize v2014.2.15 sources directly off your site) and I can happily confirm it is indeed working flawlessly! Thanks again for your help!


Thu Oct 16, 2014 22:21
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 20 posts ] 


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.