FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Wed May 12, 2021 01:47



Post new topic Reply to topic  [ 7 posts ] 
special request for creating new files with known size 
Author Message

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post special request for creating new files with known size
Hi,

I have a special use of the creating new file, the file size is already known, and the data of the file should be continuously stored in the disk space, so, when I read the file, I can directely read from the disk sectors after caculating the starting LCN from the run list.
Any advice?

Thanks.
Ryan


Tue Jan 15, 2013 08:15
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: special request for creating new files with known size
Hi,

Quote:
I have a special use of the creating new file, the file size is already known, and the data of the file should be continuously stored in the disk space

This is not possible on Linux, using the standard file access methods (the only ones which fuse can forward to ntfs-3g). On Windows you can lseek() to wanted size and write() to allocate to full size, but on Linux this will make a sparse file with no space allocated.
Quote:
Any advice?

You may develop an utility relying on libntfs-3g to do that. See for instance the defragmenter from http://jp-andre.pagesperso-orange.fr/ad ... l#download whose purpose is to allocate consecutive clusters to files. You may also examine ntfsresize which has some similar needs.

Regards

Jean-Pierre


Tue Jan 15, 2013 10:00
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: special request for creating new files with known size
Thanks for reply.

I porting the ntfs-3g project to ucos platform base on the interfaces in ntfs-3g.c
I have a rough plan about the special use, but I am not sure about some functions:
1. serach the bitmap file to find whether there is continuous large space to meet requirement
2. if yes, create a new file, now the file size is 0
3. use ntfs_attr_truncate_solid to truncate the data attribute to the file size, the size is already known

I am not sure about the step 3, whether the function can meet my requirement.
if not, do I have any chance if I modify ntfs_cluster_alloc function.
As my understanding, this function read from the bitmap to find the free clusters and then write back the bitmap and return the runlist. Here, I can write back the bitmap because I can decide the bitmap position after step 1, and , the runlist is simple, because the space is continuous.

Can I know your advice?
Thanks.


Tue Jan 15, 2013 11:40
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: special request for creating new files with known size
Hi again,

Quote:
1. serach the bitmap file to find whether there is continuous large space to meet requirement

This is the main issue. If the file system is fragmented, you are unlikely to find more than 4096 consecutive clusters.
Quote:
2. if yes, create a new file, now the file size is 0

Creating a file of size 0 generally makes the file resident, which does not reference any cluster. The main issue here is to force the new file to begin at the location defined in step 1.
Quote:
3. use ntfs_attr_truncate_solid to truncate the data attribute to the file size, the size is already known

This is probably ok if the beginning of the file is previously allocated, and it should allocate consecutive clusters when possible.
Quote:
if not, do I have any chance if I modify ntfs_cluster_alloc function.
As my understanding, this function read from the bitmap to find the free clusters and then write back the bitmap and return the runlist.

This function does not examine the whole partition, only a region of 8*4096 clusters. You will have to change that. Moreover, if you want big consecutive files, you may have to relocate existing ones.

If you know how to allocate clusters without fragmenting, that will be a major improvement.

Regards

Jean-Pierre


Tue Jan 15, 2013 13:27
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: special request for creating new files with known size
Thanks for your answer, Jean.

Now I meet a problem when alloc clusters.
The disk size is 1,991,503,872 bytes, that is, 1.85G
the free size is 1,327,681,536 bytes, that is, 1.23G

Now I need to create a file, whose size is 1,000,000,000, which needs 244,141 clusters, the alloc result is fail.
the log info is:
All zones are finished, no space on device.
At err_ret.
NTFS-fs DEBUG: Dumping runlist (values in decimal):
VCN LCN Run length
0 253952 1656
1656 288376 31112
32768 336961 15295
48063 356950 15786
63849 393953 23839
87688 419448 31112
118800 452216 31112
149912 484984 1223
151135 2 48291
199426 89717 4491
203917 162225 1615
205532 194832 26352
231884 LCN_RL_NOT_MAPPED 0 <NULL>

I trace the code, the alloc is start from data1-->data2-->mft.
And each zone info is as following, according to the log:
mft_zone_pos = 0x27915
mft_zone_start = 0x27915
mft_zone_end = 0x3667c
data1_zone_pos = 222844
data2_zone_pos = 0

From the dumping list, we can see that the free clusters of data1 zone and data2 zone are not enough for the new file size. So, also alloc from the mft zone.

According to the dumping list, it seems no cluster is alloc from the mtf zone.

what may be the reason?


Tue Jan 29, 2013 11:22
Profile

Joined: Wed Nov 23, 2011 09:11
Posts: 35
Post Re: special request for creating new files with known size
Hi Jean,

below is a piece of code from ntfs_cluster_alloc

buf_size = (int)br << 3;
lcn = bmp_pos & 7;
bmp_pos &= ~7;
writeback = 0;

while (lcn < buf_size) {
byte = buf + (lcn >> 3);
bit = 1 << (lcn & 7);
if (has_guess) {
if (*byte & bit) {
OS_PRINTF("333\n");
has_guess = 0;
break;
}
} else {
lcn = max_empty_bit_range(buf, br);
if (lcn < 0)
break;
has_guess = 1;
continue;
}

I find that if the space of zone_data1 and zone_data2 is not enough for the new file, then request space from the mft_zone. the code read the bitmap file from the position of mft zone, but unfortunately, the 1st cluster of mft zone is in use, in my case lcn = bmp_pos & 7 is 5, so the (*byte & bit) is true, then the code will treate the whole 4096*8 clusters as in using, and then break and read the next 4096 bytes in bitmap.

I think this is a bug. the has_guess should be set to 0 when switching zone from data2 to mft.
after this modification, the 1,000,000,000 byte file can successfully create


Wed Jan 30, 2013 09:05
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: special request for creating new files with known size
Hi,

Quote:
Now I need to create a file, whose size is 1,000,000,000, which needs 244,141 clusters, the alloc result is fail.
the log info is:
All zones are finished, no space on device.

This is a known issue which can happen for big allocations on small devices (see "Use of big write buffers" in param.h). The known workarounds lead to issues when several files are created concurrently.
Quote:
then the code will treate the whole 4096*8 clusters as in using, and then break and read the next 4096 bytes in bitmap.

This is intentional (space left for a possible concurrently created file), at least when allocating within a zone.
Quote:
I think this is a bug. the has_guess should be set to 0 when switching zone from data2 to mft.

You are probably right, in the special case of cross-zone allocation. More tests needed.

Regards

Jean-Pierre


Tue Feb 05, 2013 22:44
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.