FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Fri May 07, 2021 03:34



Post new topic Reply to topic  [ 3 posts ] 
Thin provisioned disks; the problem and solution 
Author Message

Joined: Wed Jun 04, 2014 17:10
Posts: 2
Post Thin provisioned disks; the problem and solution
I recently ran into an issue using ntfs-3g with thin provisioned disks. After creating a new NTFS partition and copying a large number of files to it, the thin provisioned disk had expanded to nearly its maximum size; several times the size of the actual in-use space. I tracked this down to a side effect of the cluster allocation algorithm.

Every time ntfs-3g starts a new non-resident attribute (typically the $Data attribute of a file) it offsets it 16MB from the last one. This is great for reducing file fragmentation since it leaves plenty of room for the file to expand, but creates a problem for thin provisioned disks since the files tend to be very spread out.

The fix was pretty straight forward: I changed the NTFS_LCNALLOC_SKIP value to be a mount parameter instead of a #define. By setting this parameter very small the allocation algorithm effectively optimizes for reducing disk fragmentation (all the allocated clusters are packed in one part of the disk), but at the expense of greater file fragmentation. By setting the value large the algorithm optimizes for reducing file fragmentation at the expense of disk fragmentation (which is typically not an issue, unless like me you are using thin provisioned disks.) In my particular use case I’m creating a new volume, copying files to it one at a time, and then forever after the disk is read-only. Since file fragmentation in this case is not a concern, but disk fragmentation is I set the parameter to zero.

I’m just calling it out here because this may be a feature that other users find useful.


Wed Jun 04, 2014 17:33
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Thin provisioned disks; the problem and solution
Hi,

Quote:
After creating a new NTFS partition and copying a large number of files to it, the thin provisioned disk had expanded to nearly its maximum size; several times the size of the actual in-use space. I tracked this down to a side effect of the cluster allocation algorithm.

Strange.
Quote:
Every time ntfs-3g starts a new non-resident attribute (typically the $Data attribute of a file) it offsets it 16MB from the last one. This is great for reducing file fragmentation since it leaves plenty of room for the file to expand, but creates a problem for thin provisioned disks since the files tend to be very spread out.

This is true, but does only lead to unallocatable space when the sum of the biggest holes in each set of 4096 clusters (usually 16MB) is smaller than the allocation increment. To serve an allocation request, the current allocation policy allocates a single run (the biggest one available) per set of 4096 clusters.

If you got an early "no more space", you probably allocated a lot of clusters in a single allocation request. See in include/ntfs-3g/param.h why big_writes is disabled on small disks.

Rather than using a mount option to define the allocation gap, I would be more interested in defining some relation between the gap size and the disk capacity. The current gap is not big enough for partitions holding videos, and when some gap size has been in use, it leads to a fragmentation pattern which makes it useless to switch to a different gap size for subsequent mounts.

Note : ntfscp has an option -m which minimizes the fragmentation.

Regards

Jean-Pierre


Wed Jun 04, 2014 21:33
Profile

Joined: Wed Jun 04, 2014 17:10
Posts: 2
Post Re: Thin provisioned disks; the problem and solution
In my case the issue is not that the disk is running out of space prematurely, but rather that it is consuming more space than expected on the backing storage. I’m using a thin provisioned disk; specifically a “dynamically expanding” virtual disk on a Hyper-V virtual machine. I initially create a 100GB disk, but because it is thin provisioned it only consumes a tiny bit of the actual backing storage (in this case, 4MB). The way Hyper-V thin provisioning works, every time a new 32MB aligned chunk of the disk is written to, the VHDX file backing the disk grows by 32MB. Ideally the file backing this virtual disk should only grow to a size equal to the amount of space actually in use. Chunks of the disk not yet written consume no space on the backing storage.

However by spreading files all around the disks near every one of the 32MB chunks of the thin provisioned disk is written to. I found by writing just 20GB of data to the 100GB virtual disk caused the backing file to grow to 92GB. After setting NTFS_LCNALLOC_SKIP to zero I got the result I hoped for, writing 20GB of data to the 100GB virtual disk caused the backing file to grow to only 20GB.


Thu Jun 05, 2014 15:30
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 3 posts ] 


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.