FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Fri Jun 18, 2021 20:27



Post new topic Reply to topic  [ 12 posts ] 
problems with dedup beta plugin 
Author Message

Joined: Wed Aug 24, 2016 23:26
Posts: 12
Post problems with dedup beta plugin
Hi,
I tried ntfs-3g 2016.2.22AR.2 with dedup beta plugin (compiled it from source, the supplied binary .so has the same problem).
Used Windows 2012R2 to enable dedup on some volume, put some files and after clean Windows shutdown try to mount it with ntfs-3g.
Without dedup plugin, the files that were actually deduped are shown as unsupported reparse points, and it's OK.
With dedup plugin, the files are shown in ls -l with correct sizes, i.e. stat works, but I can't read them.
dedup_read triggers "There is a gap between 0 and some_address in file inode", i.e previous_end is 0.
I'd be very thankful if somebody could help me with this.
Of course, if needed I can beta-test, play with src etc if a cooperation from my side helps.

Thanks much,

Maxim.


Fri Nov 04, 2016 03:12
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: problems with dedup beta plugin
Hi,

Quote:
With dedup plugin, the files are shown in ls -l with correct sizes, i.e. stat works, but I can't read them.
dedup_read triggers "There is a gap between 0 and some_address in file inode", i.e previous_end is 0.

Is this a general behavior, or do you get this only on specific files ?
Quote:
I'd be very thankful if somebody could help me with this.

I will try to fix this, but I first need to replicate this behavior.

First can you select a problematic file, from a device which is not to be updated soon (updating a deduplicated file interferes with other files on the same device), and please select a file not too big, preferably one whose contents can be disclosed publicly (at least disclosed to me...).

I will have to step through the data (see http://jp-andre.pagesperso-orange.fr/co ... uplication), and the first step is getting the reparse data.

Please post the output of :
Code:
getfattr -e hex -n system.ntfs_reparse_data problematic-file

Note : no user contents is disclosed at this stage.

Also, please post the file list of the deduplication directory. Again, there is no user contents disclosed at this stage (replace MOUNTPOINT by the relevant mountpoint).
Code:
ls -lR MOUNTPOINT/'System Volume Information'/Dedup | grep ccc$

Note : you have to escape the spaces in 'System Volume Information'.

When I get this, I can determine which other files I need (two more steps).

Regards

Jean-Pierre


Fri Nov 04, 2016 09:46
Profile

Joined: Wed Aug 24, 2016 23:26
Posts: 12
Post Re: problems with dedup beta plugin
Hi Jean-Pierre,
Thanks much for your help and sorry for the delay.
I reproduced the problem at my equipment I can publicly post the data from.
The problem did not occur with Windows 2016 dedup, but occurs with Windows 2012R2 dedup.
When the problem occurs, it occurs for every deduped file (seen as reparse point with permissions 555 while regular files are seen with 777).
I have an image of 500MB partition used for tests and I mount it read only.
With 80000013 plugin enabled (compiled from source), I doubled the log message to appropriate tty.
# cp "/mnt/Program Files (x86)/Windows NT/TableTextService/TableTextServiceYi.txt" /dev/null
There is a gap between 0x0 and 0x1000000000042 in file 205
There is a gap between 0x0 and 0x1000000000042 in file 205
cp: error reading '/mnt/Program Files (x86)/Windows NT/TableTextService/TableTextServiceYi.txt': Operation not supported

# ls -li /mnt/Program\ Files/Far\ Manager/Plugins/Network
total 167
791 -rwxrwxrwx 1 root root 8032 Sep 2 15:10 changelog
792 -rwxrwxrwx 1 root root 3292 Sep 2 15:10 NetEng.hlf
793 -rwxrwxrwx 1 root root 1428 Sep 2 15:10 NetEng.lng
794 -rwxrwxrwx 1 root root 5394 Sep 2 15:10 NetRus.hlf
795 -rwxrwxrwx 1 root root 2703 Sep 2 15:10 NetRus.lng
796 -rwxrwxrwx 1 root root 1578 Sep 2 15:10 NetSky.lng
797 -rwxrwxrwx 1 root root 1597 Sep 2 15:10 NetSpa.lng
798 -r-xr-xr-x 1 root root 133632 Sep 2 15:10 Network.dll
# strace cp /mnt/Program\ Files/Far\ Manager/Plugins/Network/Network.dll /dev/null 2>&1 | egrep "/mnt|read.*= -"
execve("/bin/cp", ["cp", "/mnt/Program Files/Far Manager/P"..., "/dev/null"], [/* 24 vars */]) = 0
stat("/mnt/Program Files/Far Manager/Plugins/Network/Network.dll", {st_mode=S_IFREG|0555, st_size=133632, ...}) = 0
open("/mnt/Program Files/Far Manager/Plugins/Network/Network.dll", O_RDONLY) = 3
There is a gap between 0x0 and 0x10000000000a1 in file 798
There is a gap between 0x0 and 0x10000000000a1 in file 798
There is a gap between 0x0 and 0x10000000000a1 in file 798
read(3, 0x7fb364896000, 131072) = -1 EOPNOTSUPP (Operation not supported)
write(2, "error reading '/mnt/Program File"..., 74error reading '/mnt/Program Files/Far Manager/Plugins/Network/Network.dll') = 74

# getfattr -e hex -n system.ntfs_reparse_data /mnt/Program\ Files/Far\ Manager/Plugins/Network/Network.dll
getfattr: Removing leading '/' from absolute path names
# file: mnt/Program Files/Far Manager/Plugins/Network/Network.dll
system.ntfs_reparse_data=0x130000807c00000001027c00000000000 \
00a0200000000000000000000000000 \
1c61df8786f1834b9dbd58c3123901b7 \
4000400040000000610c1a725c39d201 \
a100000000000100c065010001000000 \
0100000048010000c800000000000000 \
62a41c03a5293cc5a4d2771f49b1ede8 \
58dbfebcd6501e3d892d51e3345f8d7d \
38c3ab2d

# ls -lR /mnt/System\ Volume\ Information | egrep "ccc$"
-rwxrwxrwx 1 root root 48353280 Nov 8 03:07 00000001.00000001.ccc
-rwxrwxrwx 1 root root 10506240 Nov 8 03:07 00010000.00000001.ccc

# find /mnt/System\ Volume\ Information | egrep "ccc$"
/mnt/System Volume Information/Dedup/ChunkStore/{87DF611C-F186-4B83-9DBD-58C3123901B7}.ddp/Data/00000001.00000001.ccc
/mnt/System Volume Information/Dedup/ChunkStore/{87DF611C-F186-4B83-9DBD-58C3123901B7}.ddp/Stream/00010000.00000001.ccc

Please tell how to proceed and how I can help.

Thanks much,

Maxim.


Tue Nov 08, 2016 09:43
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: problems with dedup beta plugin
Hi,

Quote:
The problem did not occur with Windows 2016 dedup, but occurs with Windows 2012R2 dedup.

My own tests were made using Windows 2012 deduplication which could be at some time installed on Windows 10, but this is not possible any more.
Code:
# file: mnt/Program Files/Far Manager/Plugins/Network/Network.dll
system.ntfs_reparse_data=0x130000807c00000001027c000000000

This reparse data is much different (and simpler) from my own. Either this is an older version or the simpler version is used when some condition is met.
Quote:
deduped file (seen as reparse point with permissions 555 while regular files are seen with 777).

This is intentional, to prevent files with unknown format from being modified.
Quote:
# ls -lR /mnt/System\ Volume\ Information | egrep "ccc$"
-rwxrwxrwx 1 root root 48353280 Nov 8 03:07 00000001.00000001.ccc
-rwxrwxrwx 1 root root 10506240 Nov 8 03:07 00010000.00000001.ccc

As you only have a couple of files, this is an easy example. Maybe this is why you have simpler reparse data. Also please stick to Network.dll whose size is easy to manage.
From what I gather, the part I am interested in now is within Stream/00010000.00000001.ccc from offset 0x165c0 and 12800 bytes long.
For instance you can :
Code:
dd if=STREAM skip=5724 bs=16 count=800 | gzip > stream.sample.gz

Where STREAM is the stream file name (ending with Stream/00010000.00000001.ccc).

Quote:
Please tell how to proceed and how I can help.

Just post the file stream.sample.gz as an attachment to your reply (this forum only accepts compressed files as attachments). This will disclose no user data.

Regards

Jean-Pierre

PS please read again the dd command before posting. There was an error and I had to edit it.


Wed Nov 09, 2016 10:24
Profile

Joined: Wed Aug 24, 2016 23:26
Posts: 12
Post Re: problems with dedup beta plugin
Hi Jean-Pierre!
Thank you for the answer.
Actually the reparse data is not so short, the lines just were folded as I didn't use code tag.
Code:
# getfattr -e hex -n system.ntfs_reparse_data /mnt/Program\ Files/Far\ Manager/Plugins/Network/Network.dll
getfattr: Removing leading '/' from absolute path names
# file: mnt/Program Files/Far Manager/Plugins/Network/Network.dll
system.ntfs_reparse_data=0x130000807c00000001027c0000000000 \
000a0200000000000000000000000000 \
1c61df8786f1834b9dbd58c3123901b7 \
4000400040000000610c1a725c39d201 \
a100000000000100c065010001000000 \
0100000048010000c800000000000000 \
62a41c03a5293cc5a4d2771f49b1ede8 \
58dbfebcd6501e3d892d51e3345f8d7d \
38c3ab2d

Here is the data you asked for, attaching it.
Code:
# dd if="/mnt/System Volume Information/Dedup/ChunkStore/{87DF611C-F186-4B83-9DBD-58C3123901B7}.ddp/Stream/00010000.00000001.ccc" skip=5724 bs=16 count=800 | gzip > stream.sample.gz
800+0 records in
800+0 records out
12800 bytes (13 kB, 12 KiB) copied, 0.00755549 s, 1.7 MB/s
# ls -l stream.sample.gz
-rw-r--r-- 1 root root 8774 Nov 10 00:44 stream.sample.gz


Thanks much,

Maxim.


Attachments:
stream.sample.gz [8.57 KiB]
Downloaded 728 times
Thu Nov 10, 2016 00:50
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: problems with dedup beta plugin
Hi,

Quote:
Actually the reparse data is not so short, the lines just were folded as I didn't use code tag.

Your initial post was correct, I just had to reformat it as long lines are unreadable (and I have reformatted again).
Quote:
Here is the data you asked for, attaching it.

Thanks. It has the expected contents.

Now the last step : I now need an extract from the file whose name ends in Data/00000001.00000001.ccc and this will contain user data. However this will only contain parts from Network.dll (which may be duplicated in other files).

The part I need starts at offset 0x1cc7088 and ends at 0x1cda502. This is about 60% of the original file, so it is compressed. You can extract it by :
Code:
dd if=DATA seek=1885960 bs=16 count=5000 | gzip > data.sample.gz

Replace DATA by the file whose name ends in Data/00000001.00000001.ccc
For posting as an attachment you have to use gzip, though it is unlikely to be able to compress. As a consequence the file is unlikely to fit into the allowance and you may have to split it into two parts and reply twice :
Code:
dd if=DATA seek=1885960 bs=16 count=2500 | gzip > data1.sample.gz
dd if=DATA seek=1888460 bs=16 count=2500 | gzip > data2.sample.gz

If you cannot do it that way, you will have to upload the file to a public server.

Regards

Jean-Pierre


Thu Nov 10, 2016 10:17
Profile

Joined: Wed Aug 24, 2016 23:26
Posts: 12
Post Re: problems with dedup beta plugin
Hi Jean-Pierre,
Thank you, let's try to upload the contents.
Code:
# dd if="/mnt/System Volume Information/Dedup/ChunkStore/{87DF611C-F186-4B83-9DBD-58C3123901B7}.ddp/Data/00000001.00000001.ccc" seek=1885960 bs=16 count=5000 > data.sample
5000+0 records in
5000+0 records out
80000 bytes (80 kB, 78 KiB) copied, 0.00699055 s, 11.4 MB/s
# gzip data.sample
# ls -l data*
-rw-r--r-- 1 root root 86393 Nov 10 11:42 data.sample.gz


Thanks much,

Maxim.


Thu Nov 10, 2016 11:47
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: problems with dedup beta plugin
Hi,

Code:
# dd if="/mnt/System Volume Information/Dedup/ChunkStore/{87DF611C-F186-4B83-9DBD-58C3123901B7}.ddp/Data/00000001.00000001.ccc" seek=1885960 bs=16 count=5000

My fault. The input filtering is option skip, not seek.
Please retry with :
Code:
# dd if="/mnt/System Volume Information/Dedup/ChunkStore/{87DF611C-F186-4B83-9DBD-58C3123901B7}.ddp/Data/00000001.00000001.ccc" skip=1885960 bs=16 count=5000

(I will delete the bad attachment, by the way it proves the forum accepts a 80K attachment).

Regards

Jean-Pierre


Thu Nov 10, 2016 12:09
Profile

Joined: Wed Aug 24, 2016 23:26
Posts: 12
Post Re: problems with dedup beta plugin
Sorry, Jean-Pierre, didn't look either.
Code:
# dd if="/mnt/System Volume Information/Dedup/ChunkStore/{87DF611C-F186-4B83-9DBD-58C3123901B7}.ddp/Data/00000001.00000001.ccc" skip=1885960 bs=16 count=5000 | gzip > new_data.sample.gz
5000+0 records in
5000+0 records out
80000 bytes (80 kB, 78 KiB) copied, 0.0282472 s, 2.8 MB/s
# ls -l new_data.sample.gz
-rw-r--r-- 1 root root 72818 Nov 10 12:22 new_data.sample.gz


Thanks much,

Maxim.


Attachments:
new_data.sample.gz [71.11 KiB]
Downloaded 709 times
Thu Nov 10, 2016 12:25
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: problems with dedup beta plugin
Hi,

Fine ! I can get a consistent Network.dll, with an md5 of :
Code:
686362a08c004ee68e22929025510469  Network.dll

Please find a (temporary) patch to dedup.c attached, you apparently know how to compile it.

I still do not know why there are at least two formats for the reparse data, so please examine more files for other possible variations.

Regards

Jean-Pierre


Attachments:
dedup.c.patch.gz [1.74 KiB]
Downloaded 738 times
Thu Nov 10, 2016 13:56
Profile

Joined: Wed Aug 24, 2016 23:26
Posts: 12
Post Re: problems with dedup beta plugin
Hi Jean-Pierre!
Thank you very much.
Your patch works like a charm, both on Windows 2012 and Windows 2016 dedup.
All deduped exe/dll/sys files are read correctly.
When do you plan to update dedup.zip on your site with your patch, please?
I see your dedup.c is a bit newer as the patch succeeds with offsets of ~25 lines.

Thanks much,

Maxim.


Fri Nov 11, 2016 00:37
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: problems with dedup beta plugin
Hi,

Quote:
Your patch works like a charm, both on Windows 2012 and Windows 2016 dedup.

Great ! and thanks for your help.
Quote:
When do you plan to update dedup.zip on your site with your patch, please?

Done now, its md5 is bec8ceab5cc877ee6ba1fcf3922d9547
Remember : you need at least ntfs-3g-2016.2.22AR.1 to make use of it.
Quote:
I see your dedup.c is a bit newer as the patch succeeds with offsets of ~25 lines.

Not really, these 25 lines are related to debugging tools (non public).

Regards

Jean-Pierre


Fri Nov 11, 2016 09:40
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.