FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Thu May 13, 2021 01:58



Post new topic Reply to topic  [ 10 posts ] 
NTFS-3G lists but can't access files 
Author Message

Joined: Fri Jun 05, 2009 00:52
Posts: 15
Post NTFS-3G lists but can't access files
I reported this problem a while ago in the old forums. After I trashed my NTFS partition, I had tons of files that NTFS-3G could not read. Any attempt to access them gave "File not found" errors, yet `ls` directory listings showed them fine. And the testdisk utility had no problem copying these files.

I fixed my partition (by booting from a slipstreamed nLite'd XP recovery disk with SATA drivers in order to run CHKDSK), but before doing so I backed up the partition. I think it's awful behavior of ntfs-3g to not show files to the user that it knows are there, so I'm returning to this problem in the hope of getting ntfs-3g improved. Back in June 2009 I and jpa "Jean-Pierre" thought the problem might arise from messed up file attributes, but now I think it might be the volume collation...

I've compiled the latest source and can reproduce at will, but I'm not sure what else to do. Here are the commands to access a file and the output from mount.ntfs-3g that they engender:
Code:
$ sudo sbin/mount.ntfs-3g -o ro,debug /dev/sdb3 ~/mnt_testNTFS
$ cat ~/mnt_testNTFS/boot.ini

LOOKUP /boot.ini
ntfs_pathname_to_inode(): path: '/boot.ini'
ntfs_inode_lookup_by_name(): Entering
ntfs_attr_lookup(): Entering for attribute type 0x90
ntfs_attr_find(): attribute type 0x90.
ntfs_attr_open(): Entering for inode 5, attr 0xa0.
ntfs_attr_lookup(): Entering for attribute type 0xa0
  ntfs_attr_find(): attribute type 0xa0.
ntfs_attr_mst_pread(): Entering for inode 0x5, attr type 0xa0, pos 0x0.
ntfs_attr_pread(): Entering for inode 5 attr 0xa0 pos 0 count 4096
ntfs_attr_find_vcn(): Entering for inode 0x5, attr 0xa0, vcn 0
ntfs_attr_map_runlist(): Entering for inode 0x5, attr 0xa0, vcn 0x0.
ntfs_attr_lookup(): Entering for attribute type 0xa0
  ntfs_attr_find(): attribute type 0xa0.
ntfs_mapping_pairs_decompress(): Entering
  ntfs_mapping_pairs_decompress_i(): Entering for attr 0xa0.
  Mapping pairs array successfully decompressed:
  NTFS-fs DEBUG: Dumping runlist (values in hex):
  VCN              LCN               Run length
  0                7511784           2
  2                LCN_ENOENT        0                (runlist end)
ntfs_attr_pread_i(): Reading 4096 bytes from vcn 0, lcn 7511784, ofs 0.
ntfs_pread(): pos 30768267264, count 4096
ntfs_mst_post_read_fixup(): Entering
Entry not found.
Couldn't find name 'boot.ini' in pathname '/boot.ini'.
   unique: 774, error: -2 (No such file or directory), outsize: 16

Using gdb, I determined the "Entry not found." is the second such debug message in libntfs-3g/dir.c's ntfs_inode_lookup_by_name() (BUG: the two ntfs_log_debug() messages should be distinct!). The stack trace is
Code:
#0  ntfs_inode_lookup_by_name (dir_ni=0x61c390, uname=0x621990, uname_len=8) at dir.c:519
#1  0x00007ffff7b9ce3d in ntfs_pathname_to_inode (vol=0x617b40, parent=0x0,
    pathname=<value optimized out>) at dir.c:781
#2  0x00000000004065e6 in ntfs_fuse_getattr (org_path=<value optimized out>,
    stbuf=<value optimized out>) at ntfs-3g.c:698
#3  0x00007ffff7bbb338 in lookup_path (f=0x620a70, nodeid=1, name=0x7ffff7e36038 "boot.ini",
    path=0x8 <Address 0x8 out of bounds>, e=0x102, fi=0x7ffff7e58010) at fuse.c:943
#4  0x00007ffff7bbcfb3 in fuse_lib_lookup (req=0x620780, parent=<value optimized out>,
    name=0x7ffff7e36038 "boot.ini") at fuse.c:1100
#5  0x00007ffff7bbf72a in fuse_session_loop (se=0x620fc0) at fuse_loop.c:34
#6  0x00000000004082aa in main (argc=<value optimized out>, argv=<value optimized out>)
    at ntfs-3g.c:4496


The interesting thing is if I'm understanding it right, ntfs_inode_lookup_by_name() starts its first loop through index entries, calls ntfs_names_full_collate() to compare uname "boot.ini" with the first entry "Documents and Settings" , and exits the loop! The comment at libntfs-3g/dir.c line 335 being If uname collates before the name of the current entry, there is definitely no such name in this index. So then it goes on to the second loop that goes through this child node, finds nothing of interest, and prints "Entry not found."

I'm guessing and am way out of my depth, but perhaps the vol->upcase table for this NTFS volume is messed up. upcase[98] (the 'b' in "boot.ini") is 51673, upcase[98] (the 'D' in "Documents and Settings") is also 51673. That doesn't seem right. Are there ways to examine, repair, or override the volume's $UpCase table?

I can't give you my disk but I would be happy to step through this and run any code or gdb commands to puzzle out what's going on! Thanks indeed.


Tue Sep 28, 2010 12:01
Profile

Joined: Fri Jun 05, 2009 00:52
Posts: 15
Post Re: NTFS-3G lists but can't access files
Yeah, I think it's a corrupt $UpCase file on my garbled partition. If I skip over the part of ntfs_device_mount() in in libntfs-3g/volume.c that reads it in, then the volume has the default upcase table , and then ntfs-3g gets collation right and ntfs_inode_lookup_by_name() succeeds. That means I can access files on this corrupted partition, can run `ls -l` without choking, etc.! :P Maybe the testdisk utility that has been able to read all the files fine doesn't attempt to read in $UpCase.

So I repeat my earlier questions: Are there ways to examine, repair, or override an NTFS volume's $UpCase table? And/or, maybe there could be an option to ntfs-3g to ignore the $UpCase table and use its default one.

I don't know if there could be a sanity check for the volume's $UpCase file to detect that it's corrupt. Mine had garbage in it, e.g. it mapped several low ASCII symbols like '$' to crazy high numbers, but maybe that's appropriate in some encodings?

BUG: ntfs-3g goes to the trouble of creating a default vol->upcase table, but to my eyes it seems it never uses it. If anything goes wrong in reading in $UpCase and replacing the default table, ntfs-3g fails to mount the volume.

I hope this helps someone, it's been interesting.


Tue Sep 28, 2010 13:48
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G lists but can't access files
Hi,

Quote:
I'm guessing and am way out of my depth, but perhaps the vol->upcase table for this NTFS volume is messed up. upcase[98] (the 'b' in "boot.ini") is 51673, upcase[98] (the 'D' in "Documents and Settings") is also 51673. That doesn't seem right. Are there ways to examine, repair, or override the volume's $UpCase table?

These upcase values are definitely wrong. upcase['b'] should be 'B' and upcase['D'] should be 'D'.
Quote:
Yeah, I think it's a corrupt $UpCase file on my garbled partition. If I skip over the part of ntfs_device_mount() in in libntfs-3g/volume.c that reads it in, then the volume has the default upcase table , and then ntfs-3g gets collation right and ntfs_inode_lookup_by_name() succeeds.

This confirms that you have an upcase table issue. Note that ntfs-3g never writes the upcase table. This can only result from another error such as a bad allocation table. Do you know how the issue arose (ntfs-3g version) ?
Quote:
So I repeat my earlier questions: Are there ways to examine, repair, or override an NTFS volume's $UpCase table?

chkdsk does write on the designated volume the upcase table for the current Windows system (there are differences between OSes).
Quote:
And/or, maybe there could be an option to ntfs-3g to ignore the $UpCase table and use its default one.

There could be such an option, with the consequence that some files created with the wrong table might become unreadable (for instance if "boot.ini" were located after "Documents and Settings").
Quote:
I don't know if there could be a sanity check for the volume's $UpCase file to detect that it's corrupt.

This could only be done for some character subset. Without a checksum, ntfs-3g cannot recognize a wrong value from a desired one.

Regards

Jean-Pierre


Tue Sep 28, 2010 14:51
Profile

Joined: Fri Jun 05, 2009 00:52
Posts: 15
Post Re: NTFS-3G lists but can't access files
Thanks so much for your rapid attentive response!
jpa wrote:
Note that ntfs-3g never writes the upcase table. This can only result from another error such as a bad allocation table. Do you know how the issue arose (ntfs-3g version) ?
I overwrote the first GB or so of my hard drive using Cygwin dd command! (Folks, be careful with those sda's and sdb's :evil: )

jpa wrote:
Quote:
And/or, maybe there could be an option to ntfs-3g to ignore the $UpCase table and use its default one.
There could be such an option, with the consequence that some files created with the wrong table might become unreadable (for instance if "boot.ini" were located after "Documents and Settings").
So an ignoreupcase option to mount.ntfs would be an advanced feature, and as I mentioned I think it's the only way ntfs-3g would ever actually use the default UpCase table that it lovingly builds.

jpa wrote:
Quote:
I don't know if there could be a sanity check for the volume's $UpCase file to detect that it's corrupt.
This could only be done for some character subset. Without a checksum, ntfs-3g cannot recognize a wrong value from a desired one.
It's unfortunate that a file on disk can make the index walking code fail to find files.
  • Why can Windows and the Linux testdisk utility still find files when $UpCase is corrupt?
Testdisk seems to use ntfsprogs, but ntfscat seems unable to copy files that testdisk can; I guess they use different paths.

I looked at ntfsprogs and it seems their tools don't do anything with $UpCase. E.g. mkfs.ntfs has no option for character set, ntfsfix doesn't seem to fix it. ntfsprogs development seems stalled, so is there any point in making feature requests for these programs?

Anyway, thanks again for your expertise and this fine software.
Anyone watching, take the time to build both a bootable Windows rescue CD/USB and a bootable Linux rescue CD/USB containing testdisk and other disk utilities!


Tue Sep 28, 2010 22:59
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G lists but can't access files
Hi,

Quote:
It's unfortunate that a file on disk can make the index walking code fail to find files.

This must be the case in most file systems....
Quote:
Why can Windows and the Linux testdisk utility still find files when $UpCase is corrupt?

The Upcase file is essential to locate a file by its name in a Btree, but for accessing a file by its inode number the Upcase file is not needed.
Quote:
I looked at ntfsprogs and it seems their tools don't do anything with $UpCase. E.g. mkfs.ntfs has no option for character set, ntfsfix doesn't seem to fix it.

Attached is a patch to ntfsfix for replacing the current Upcase file by a standard one which is similar to the one created by WinXP (Vista and probably Win7 use a slightly different one). Use the option -u to fix the Upcase file.
Quote:
ntfsprogs development seems stalled, so is there any point in making feature requests for these programs?

So you have to implement them yourself.

Regards

Jean-Pierre


Attachments:
ntfsprogs-upcase.patch.gz [2.32 KiB]
Downloaded 833 times
Sat Oct 02, 2010 10:39
Profile

Joined: Fri Jun 05, 2009 00:52
Posts: 15
Post ntfsprogs source code, issue summary
Thanks for your reply and explanations, I either overlooked or didn't receive notification of it.

jpa wrote:
Attached is a patch to ntfsfix for replacing the current Upcase file by a standard one which is similar to the one created by WinXP (Vista and probably Win7 use a slightly different one). Use the option -u to fix the Upcase file.

Sounds great, but it doesn't apply to my old source copy of ntfsprogs from 2009. I get "The server at http://www.linux-ntfs.org is taking too long to respond." errors, is the code mirrored somewhere else?

Below are the issues I would file if I could find a bugbase for NTFS-3G.

This patch seems like a great fix, the downside is you have to know to apply it! As you explained earlier, NTFS-3G can't tell $UpCase is corrupt, so the user has to guess that the reason she can list but not examine most files is a corrupt $UpCase file.

Enhancement I believe it would be useful to have a test in volume.c around line 1029 after reading $UpCase, something like:
Code:
/* Sanity check $UpCase table */
if (vol->upcase_len > (u32)'A') && =vol->upcase['a'] != 'A') {
  ntfs_log_warning("The volume's $UpCase file may be corrupt: uppercase of 'a' is not 'A'.");
}


I can't list or delete '$UpCase', probably because accessing it requires it to not be corrupt :lol: . However, I stepped through the code in gdb and forced ntfs_inode_open(vol, FILE_UpCase); to return 0, and that just results in "Failed to open inode FILE_UpCase: No such file or director" and then a failed mount. So if the user deletes the corrupt $UpCase (maybe in another tool like testdisk) it won't help, and as I suspected
Bug: NTFS-3G never(?) uses the default vol->upcase table it constructs in ntfs_upcase_table_build().

I don't know what the right behavior in the face of problems reading $UpCase is, maybe rather than add an ignoreupcaseerrors mount option, just continue to print errors and in the doc suggest people run your new `ntfsfix -u` code.

Also in case you missed it, I mentioned earlier
Bug: the two ntfs_log_debug("Entry not found") messages in libntfs-3g/dir.c's ntfs_inode_lookup_by_name() should be distinct.

jpa wrote:
Quote:
ntfsprogs development seems stalled, so is there any point in making feature requests for these programs?

So you have to implement them yourself.
If you're talking to me, I wish I was better at coding. Are you a developer of ntfsprogs?

So many thanks for your help.


Sat Oct 30, 2010 08:05
Profile

Joined: Fri Jun 05, 2009 00:52
Posts: 15
Post patch to volume.c around line 1029
D'oh, the patch to volume.c around line 1029 is something like:
Code:
/* Sanity check $UpCase table */
if (vol->upcase_len > (u32)'a') && =vol->upcase['a'] != 'A') {
  ntfs_log_warning("The volume's $UpCase file may be corrupt: uppercase of 'a' is not 'A'.");
}


Sat Oct 30, 2010 08:08
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G lists but can't access files
Hi,

Quote:
Sounds great, but it doesn't apply to my old source copy of ntfsprogs from 2009. I get "The server at http://www.linux-ntfs.org is taking too long to respond." errors, is the code mirrored somewhere else?

Development of ntfsprogs seems to have stopped by 2007. If you have a more recent copy, it is the right one. There are copies made available by most distributions. The mirrored copy I use is from ftp://ftp.free.fr/mirrors/fedora.redhat ... 13.src.rpm It contains a few patches from Fedora.
Quote:
Enhancement I believe it would be useful to have a test in volume.c around line 1029 after reading $UpCase, something like:

This has been done for ntfs-3g-2010.10.2AR.1 checking plain ascii chars (codes 0x20..0x7f). See http://www.tuxera.com/community/ntfs-3g ... /#download
Quote:
Bug: NTFS-3G never(?) uses the default vol->upcase table it constructs in ntfs_upcase_table_build().

The table is there for fixing tools to use it. ntfs-3g must use the table on the partition, which depends one the OS used to format it.
Quote:
I don't know what the right behavior in the face of problems reading $UpCase is,

When ntfs-3g finds something unexpected, the policy is to stop and not write anything. Fixing is left to dedicated tools.
Quote:
Bug: the two ntfs_log_debug("Entry not found") messages in libntfs-3g/dir.c's ntfs_inode_lookup_by_name() should be distinct.

I put this on my todo list (low priority)
Quote:
If you're talking to me, I wish I was better at coding. Are you a developer of ntfsprogs?

Well, you have entered ntfs-3g with gdb (which I have never done myself), you could surely have improved ntfsfix. I had no previous knowledge of the internals of ntfsfix myself.

Regards

Jean-Pierre


Sat Oct 30, 2010 08:46
Profile

Joined: Fri Jun 05, 2009 00:52
Posts: 15
Post Re: NTFS-3G lists but can't access files
jpa wrote:
The mirrored copy I use is from ftp://ftp.free.fr/mirrors/fedora.redhat ... 13.src.rpm It contains a few patches from Fedora.
Thanks. I'll have to learn how to build from a srpm (I'm on Kubuntu).

jpa wrote:
This has been done for ntfs-3g-2010.10.2AR.1 checking plain ascii chars (codes 0x20..0x7f). See http://www.tuxera.com/community/ntfs-3g ... /#download

I tried it, excellent! It resulted in:
Code:
Corrupted file $UpCase: No such file or directory
Failed to mount '/dev/sdc3': No such file or directory
The "No such file or directory" is misleading. It seems the ntfs-3g error logger always prints ":", strerror(errno) even if errno is not relevant to the current error.


Sun Oct 31, 2010 13:28
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: NTFS-3G lists but can't access files
Hi,
Quote:
Thanks. I'll have to learn how to build from a srpm (I'm on Kubuntu).

There must also be an equivalent ntfsprogs source on Ubuntu mirrors.

To extract rpm contents, I use
Code:
rpm2cpio <rpm-file> | cpio -idm

rpm2cpio being part of the rpm package, (also within the file-roller package).

Quote:
The "No such file or directory" is misleading. It seems the ntfs-3g error logger always prints ":", strerror(errno) even if errno is not relevant to the current error.

The problem here is that ntfs-3g can only use one of the codes defined in errno.h and these limited error codes cannot describe properly what may happen in a driver. So the detailed error message is to be found in the system log. In your situation the message "Corrupted file $UpCase" must have been logged.

** edit **

There is actually a logging bug : the "No such file or directory" should not have been logged. Will be fixed.

Regards

Jean-Pierre


Tue Nov 02, 2010 18:29
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.