FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Fri May 07, 2021 03:19



Post new topic Reply to topic  [ 12 posts ] 
Some Files missed when creating files in a DIR 
Author Message

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Some Files missed when creating files in a DIR
Hi,

I ported ntfs-3g-2011.4.12 for my embedded device. But I meet problem while creating files in one DIR.
I find that some files are missed if the number of files created in DIR is larger than a specific number.

I do my operations in following steps.
1. 30 files are created in the DIR names PVRS, and all files can be recognized on windows PC. As the below PIC.
Attachment:
File comment: 30 old files can be recognized
OLD.JPG
OLD.JPG [ 113.89 KiB | Viewed 25833 times ]


At this step, the index entry of the 30 files use 2 index blocks of the "PVRS" DIR.
The index_length of index block 0 is 0x0D28, and 0x0F68 for index block 1.
2. 2 files are created, whose name are CETV1-2014-09-02 13-55-16.ridx and CETV1-2014-09-02 13-55-16.ts
But after this step, some files missed, shown in the following PIC.
Attachment:
File comment: Files missed
DIFF.JPG
DIFF.JPG [ 205.15 KiB | Viewed 25833 times ]


After debug, I find out that, when creating files, ntfs_ie_add are called. In this function, new size of the index block is calculated to see if it is larger than the allocated size(which is 0x0FE8) or not. If the new size is larger than 0x0FE8, then the index block should be split into 2 blocks from the median index entry.

In this example, index entry of file "CETV1-2014-08-27 15-43-14.ridx" is selected. The files above the index is kept in the old index block(number 0 and 1), and the below files will be moved to a new index block(number 2).

But, unfortunately, the files in index block 2 can not be recognized on windows PC. And the reason is, the index entry of the file "CETV1-2014-08-27 15-43-14.ts" should be moved to the index root attribute(0x90) of the "PVRS" MFT, but it is not.
Attachment:
File comment: MFT NOT RIGHT
PVRS_MFT.JPG
PVRS_MFT.JPG [ 213.51 KiB | Viewed 25833 times ]


And, the 2 new files created in step 2 are added into index block 1, so the 2 files can be recognized on windows PC.
But on windows, the 2 files should be add to index block 2.

I guess this is the bug of ntfs-3g-2011.4.12, but I download the latest stable version(2014.2.15), and find that the function ntfs_ie_add is still the same as the old version.

Is this a bug of ntfs-3g?
How should I modify this?
It should be work if I modify the index root attribute of MFT, and move the 2 new files to index block 2.
But this method can only add some more files to the "PVRS" DIR, due to the left size of index block 2 is limitted.
What should I do if I want to create hundreds of files in the "PVRS" DIR?

Thanks and regards.
Ryan


Wed Sep 03, 2014 04:57
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Some Files missed when creating files in a DIR
Hi,

Quote:
I ported ntfs-3g-2011.4.12 for my embedded device. But I meet problem while creating files in one DIR.
I find that some files are missed if the number of files created in DIR is larger than a specific number.

A similar issue was found in ntfs-3g-2011.4.12, it was fixed in ntfs-3g-2011.4.12AR.4 by applying the patch in http://sourceforge.net/p/ntfs-3g/ntfs-3 ... dc4c5ba97/

First of all, please apply this patch, or upgrade to a supported version.

Quote:
The files above the index is kept in the old index block(number 0 and 1), and the below files will be moved to a new index block(number 2).

There are different possibilities for splitting an index node, trying to keep the tree balanced. The new block (number 2) may become the new root and the older one made a leaf, or the new block is made a leaf and the root remains unchanged. ntfs-3g may behave differently from Windows and this is not an error.
Quote:
Is this a bug of ntfs-3g?
How should I modify this?

If the patch mentioned above is unsuccessful, please post the full index layout by ntfsinfo, so that I can replay the condition.
Code:
# as root :
ntfsinfo -fv -F path/PVRS /dev/sdxx

where path is relative to the root of the ntfs file system (NOT the root of the full file system), and sdxx is the storage device identification.
If you did not port ntfsinfo to your embedded device, you can use the Windows port from jp-andre.pagesperso-orange.fr/advanced-ntfs-3g.html
Code:
ntfsinfo -fv -F path\PVRS X:

Please do it twice : before and after the bad condition arises. Compress both outputs and post them as attachments (note : this forum requires the attachments to be compressed by gzip, bzip2, zip, etc.)

Regards

Jean-Pierre


Wed Sep 03, 2014 09:29
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: Some Files missed when creating files in a DIR
Hi Jean,

Thanks for your reply.
The patch is already in my source code.

I execute "ntfsinfo -fv -F F:\PVRS F:" and "ntfsinfo -fv -F \PVRS F:" on windows PC where the ntfs volume is F.
But this seems the wrong usage for the PC command line showes "Usage: ntfsinfo <drive letter>"
I google the usage and try lots of times but all fail.

Can I use the hex files saved by winhex instead?

Thanks.
Ryan


Wed Sep 03, 2014 11:08
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Some Files missed when creating files in a DIR
Hi,

Quote:
I execute "ntfsinfo -fv -F F:\PVRS F:" and "ntfsinfo -fv -F \PVRS F:" on windows PC where the ntfs volume is F.

On Windows, as PVRS is in the ntfs root directory you should execute (with no prefix to PVRS) :
Code:
ntfsinfo -fv -F PVRS F:

Note : you also have to use a privileged account on Windows (use "run as Administrator" if needed).

Another way is to first determine the inode number (from "ls -li PVRS" or "stat PVRS" executed on your embedded device), and execute :
Code:
ntfsinfo -fvi <inode-number> F:

According to your earlied screen capture, the inode of PVRS could be 27, so, provided you have not recreated PVRS, you can try :
Code:
ntfsinfo -fvi 27 F:

Note : ntfs-3g always creates inodes with numbers >= 32, this one has probably be created by Windows.
Quote:
Can I use the hex files saved by winhex instead?

Please spare me having to decode it all by hand !
If you cannot get ntfsinfo data, try to make a full metadata image :
Code:
ntfsclone -mst image-before F:
ntfsclone -mst image-after F:

Where image-before and image-after are files to be created (before and after the issue). As you have few files, the images should be small enough to be attached (after being compressed).

Quote:
But after this step, some files missed, shown in the following PIC

A useful clarification : are the files missing from Windows only, or are they missing from both Windows and the embedded device ?

Regards

Jean-Pierre


Wed Sep 03, 2014 14:18
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: Some Files missed when creating files in a DIR
Hi Jean,

Thanks.
The reason of the ntfsinfo failure is, the program downloaded from Microsoft website.
I setup a ubuntu system and the ntfsinfo is usefull.

I run ntfsinfo twice, before and after the condition rised. You can find it in attached before_after.zip.
BTW: the files difference is shown at the following picture.
Attachment:
diff_0904.JPG
diff_0904.JPG [ 232.73 KiB | Viewed 25812 times ]


Some files are miss on windows PC. No file is miss on my embedded board(the files can be recognized by readdir function), but the files can not be opened on the board(the same files missed on PC), the fail reason is the file can not be find(ntfs_inode_lookup_by_name return fail).

Thanks.
Ryan


Attachments:
before_after.zip [5.86 KiB]
Downloaded 832 times
Thu Sep 04, 2014 05:59
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: Some Files missed when creating files in a DIR
Re-upload the file difference picture.
Attachment:
diff_0904.JPG
diff_0904.JPG [ 220.89 KiB | Viewed 25812 times ]


Thu Sep 04, 2014 06:13
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Some Files missed when creating files in a DIR
Hi,
Quote:
The reason of the ntfsinfo failure is, the program downloaded from Microsoft website.
I setup a ubuntu system and the ntfsinfo is usefull.

It would be a big achievement for ntfs-3g to have its tools to be downloadable from Microsoft !... but are you sure you were downloading from http://jp-andre.pagesperso-orange.fr/ad ... fs-3g.html ?
Quote:
In this example, index entry of file "CETV1-2014-08-27 15-43-14.ridx" is selected. The files above the index is kept in the old index block(number 0 and 1), and the below files will be moved to a new index block(number 2).

Yes.
Quote:
But, unfortunately, the files in index block 2 can not be recognized on windows PC. And the reason is, the index entry of the file "CETV1-2014-08-27 15-43-14.ts" should be moved to the index root attribute(0x90) of the "PVRS" MFT, but it is not.

Your ntfsinfo output (after.txt) indeed shows 'CETV1-2014-08-27 15-43-14.ts' has been removed from index block 1 but not copied to index root. However the space has been reserved (152 bytes) in the index root and there was enough space for that.

When repeating the same scenario (filling the index with your file names, both long names and short names), to get a nearly full index block, then adding 'CETV1-2014-09-04 10-20-30.ridx', I get the correct behavior, and 'CETV1-2014-08-27 15-43-14.ts' is correctly moved to the index root.

I even repeated with ntfs-3g-2011.4.12 and got correct results.

Did you make changes of your own to ntfs-3g-2011.4.12 ? For instance, having two copies of the inode in the cache may lead to similar errors. Having two copies would also explain why you can readdir() all the files, though you cannot access a few ones.

The sequence for moving an entry to the index root is in ntfs_ir_insert_median() :
Code:
        ret = ntfs_ir_make_space(icx, new_size);
        if (ret != STATUS_OK)
                return ret;

        icx->ir = ntfs_ir_lookup2(icx->ni, icx->name, icx->name_len);
        if (!icx->ir)
                return STATUS_ERROR;

        return ntfs_ih_insert(&icx->ir->index, median, new_vcn,
                              ntfs_icx_parent_pos(icx));

One of these steps (reserve space, locate position, insert record) behaved badly on your system. With the attached patch, you can identify whether one of them returns an error.

Please use a 2014 version for your tests. I do not want to debug again problems which have been fixed in recent versions.

Regards

Jean-Pierre


Attachments:
index.c.patch.gz [421 Bytes]
Downloaded 781 times
Thu Sep 04, 2014 12:37
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: Some Files missed when creating files in a DIR
Hi Jean,
Quote:
It would be a big achievement for ntfs-3g to have its tools to be downloadable from Microsoft !... but are you sure you were downloading from http://jp-andre.pagesperso-orange.fr/ad ... fs-3g.html ?

This time I use the ntfsinfo installed by ubuntu itself.

Quote:
Your ntfsinfo output (after.txt) indeed shows 'CETV1-2014-08-27 15-43-14.ts' has been removed from index block 1 but not copied to index root. However the space has been reserved (152 bytes) in the index root and there was enough space for that.

When repeating the same scenario (filling the index with your file names, both long names and short names), to get a nearly full index block, then adding 'CETV1-2014-09-04 10-20-30.ridx', I get the correct behavior, and 'CETV1-2014-08-27 15-43-14.ts' is correctly moved to the index root.

I even repeated with ntfs-3g-2011.4.12 and got correct results.

Yes, the space is reserved, but filled with all zero, not the index entry of CETV1-2014-08-27 15-43-14.ts.

Quote:
Did you make changes of your own to ntfs-3g-2011.4.12 ? For instance, having two copies of the inode in the cache may lead to similar errors. Having two copies would also explain why you can readdir() all the files, though you cannot access a few ones.

I porting the ntfs-3g to my ucos platform, but do not change these basic functions dealing with index and attribute an so on.
The readdir function can recognized the files because it decode the data run area of the 0xA0 attribute, it does not need to compare the index entries with the cutting edge one(ex. the index entry of CETV1-2014-08-27 15-43-14.ts).
But if you want to open the missing file, you need to call ntfs_names_full_collate to compare the file needed to be open and the file of the cutting edge.

Quote:
The sequence for moving an entry to the index root is in ntfs_ir_insert_median() :
Code:
        ret = ntfs_ir_make_space(icx, new_size);
        if (ret != STATUS_OK)
                return ret;

        icx->ir = ntfs_ir_lookup2(icx->ni, icx->name, icx->name_len);
        if (!icx->ir)
                return STATUS_ERROR;

        return ntfs_ih_insert(&icx->ir->index, median, new_vcn,
                              ntfs_icx_parent_pos(icx));

One of these steps (reserve space, locate position, insert record) behaved badly on your system. With the attached patch, you can identify whether one of them returns an error.

Please use a 2014 version for your tests. I do not want to debug again problems which have been fixed in recent versions.

This code is the same with the original code, I do not modify this. I will find out to see which function is not right at first.
BTW, I do not see the patch you mention, did you forget to attach the patch file? Thanks.

Regards

Ryan


Fri Sep 05, 2014 03:51
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: Some Files missed when creating files in a DIR
Hi Jean,

I see the patch now.
Thanks.

Ryan


Fri Sep 05, 2014 04:07
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: Some Files missed when creating files in a DIR
Hi Jean,

I find out the root cause now.
The realloc function called by ntfs_ie_add_vcn makes this happen.
The realloc function does not copy the original memory to the new memory.

Thank you very much.

Ryan


Fri Sep 05, 2014 08:55
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Some Files missed when creating files in a DIR
Hi,

Quote:
I find out the root cause now.

Good news.
Quote:
The realloc function called by ntfs_ie_add_vcn makes this happen.
The realloc function does not copy the original memory to the new memory.

Do you mean you have your own implementation of realloc() ? ntfs-3g just uses the one provided by the clib (usually the GNU one).

Regards

Jean-Pierre


Fri Sep 05, 2014 09:06
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: Some Files missed when creating files in a DIR
Hi Jeans,

Quote:
Do you mean you have your own implementation of realloc() ? ntfs-3g just uses the one provided by the clib (usually the GNU one).


Yes, my system does not use clib but use the own implementation. I dump the index root in PVRS MFT and find the reason.
Thanks again for your help.

Regards,
Ryan


Fri Sep 05, 2014 10:00
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.