FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Tue Jun 15, 2021 10:59



Post new topic Reply to topic  [ 28 posts ]  Go to page 1, 2  Next
no space left problem when writing to USB NTFS drive 
Author Message

Joined: Tue Mar 24, 2009 14:01
Posts: 6
Post no space left problem when writing to USB NTFS drive
Hi,

I'm having problems when trying to write files into an 1 GB USB 2.0 pendrive. This unit has 968 MB of free space (5 MB are not used from the total 973 MB) and i'm trying to copy 929 MB distributed in 146 files and 8 directories. I'm not getting any error trace in /var/log/messages, /var/log/messages.log, /var/log/syslog or /var/log/daemon.log. I only get traces of mounting and umounting operations in /var/log/syslog and /var/log/daemon.log.

The problem is that the last 8 files can't be copied because the system tells that there is no free remaining space in the USB drive. Those files are touched with 0 KB in the pendrive but it has still 71 MB of free space remaining. The files that can't be copied use 32 MB of disk space.

I first believed that maybe the problem was the internal fragmentation of the disk (i don't know if this is really possible in NTFS though). Then i have noticed that when i get the error, if i umount and mount again the USB drive then, i can then copy the rest of the files without any problem.

What could the problem be?. I don't really now if this is a problem of NTFS-3G or the Linux kernel. To test this i have created a linux bash script that copies some video files and ran it through different configurations.

The system where i have tested are 32-bit GNU/Linux systems on top of Intel(R) Pentium(R) Dual CPU E2200 @ 2.20GHz:
1) Ubuntu 8.10 Intrepid Ibex (Kernel 2.6.27-11-generic #1 SMP, FUSE 2.7.3, NTFS-3G 1.2506)
2) Debian 5.0 Lenny (Kernel 2.6.26-1-686 #1 SMP, FUSE 2.7.4, NTFS-3G 1.2531)
3) Debian 5.0 Lenny (Kernel 2.6.24.4 #1 SMP, FUSE 2.7.4, NTFS-3G 2009.2.1)

The problem is reproduced always and with the same results in those systems configurations.

I have included traces from /var/log/syslog, and /var/log/daemon.log. No traces were written when copying the files or mounting the units in files /var/log/messages and /var/log/messages.log. I have also included the script i have created and the traces of running it in 2) configuration (tree command has been used to show the file sizes before and after the copying process).

Thank you very much for the support and congratulations for NTFS-3G.


Attachments:
File comment: Traces of Debian 5.0 Lenny (Kernel 2.6.26-1-686 #1 SMP, FUSE 2.7.4, NTFS-3G 1.2531) configuration.
traces.tar.gz [4.68 KiB]
Downloaded 1054 times
Tue Mar 24, 2009 16:16
Profile
Tuxera CTO

Joined: Tue Nov 21, 2006 23:15
Posts: 1648
Post Re: no space left problem when writing to USB NTFS drive
Hi,

Please download http://ntfs-3g.org/download/ntfs-3g.clone-1.5388.tgz

then run
Code:
ntfs-3g.clone -mso - DEVICE | bzip2 > DEVICE-meta.img.bz2

and send the DEVICE-meta.img.bz2 file to szaka@ntfs-3g.org.

Thanks, Szaka


Tue Mar 24, 2009 16:38
Profile

Joined: Tue Mar 24, 2009 14:01
Posts: 6
Post Re: no space left problem when writing to USB NTFS drive
szaka wrote:
Hi,

Please download http://ntfs-3g.org/download/ntfs-3g.clone-1.5388.tgz

then run
Code:
ntfs-3g.clone -mso - DEVICE | bzip2 > DEVICE-meta.img.bz2

and send the DEVICE-meta.img.bz2 file to szaka@ntfs-3g.org.

Thanks, Szaka


OK. Sent.


Tue Mar 24, 2009 16:57
Profile
Tuxera CTO

Joined: Tue Nov 21, 2006 23:15
Posts: 1648
Post Re: no space left problem when writing to USB NTFS drive
I did the following after mounting the two image files under the 'after' and 'before' directories:

Code:
cp -r after/VideoServer before
dd if=/dev/zero of=before/foo bs=1M

The copy was successful and 'dd' copied a 71 MB file until the volume became full. In short, I could not reproduce your problem.

The hardware is not relevant. One difference is that you always used the external FUSE library. NTFS-3G uses the internal one by default (it has all the officially-not-released-yet FUSE fixes, simplification, quality testing). However I'm not aware of any potentially related FUSE problem or what it could case what you experience.


Tue Mar 24, 2009 18:53
Profile
Tuxera CTO

Joined: Tue Nov 21, 2006 23:15
Posts: 1648
Post Re: no space left problem when writing to USB NTFS drive
Sorry. I was using the latest development NTFS-3G version for testing. I could reproduce the problem with the latest stable release 2009.2.1. It seems we fixed your problem even before you reported it and we knew about it ;-)

I'll check what's going on and will update.


Tue Mar 24, 2009 19:03
Profile
Tuxera CTO

Joined: Tue Nov 21, 2006 23:15
Posts: 1648
Post Re: no space left problem when writing to USB NTFS drive
Hi,

The problem was not easy to reproduce. It seems you just had the right amount of right size files on a right size NTFS volume. The development version had the same problem. It is fixed now at

http://mercurial.creo.hu/repos/ntfs-3g- ... f08d2ef030

It will be included in next stable version which is expected to be released this month. Thanks again.

Best regards, Szaka


Wed Mar 25, 2009 02:17
Profile

Joined: Tue Mar 24, 2009 14:01
Posts: 6
Post Re: no space left problem when writing to USB NTFS drive
Hi,

I will apply the patch myself meanwhile the next stable version is released.

Thank you very much again for the support and congratulations for the fantastic work.

Antonio.


Wed Mar 25, 2009 09:01
Profile

Joined: Tue Mar 24, 2009 14:01
Posts: 6
Post Re: no space left problem when writing to USB NTFS drive
Hi again,

I have modified the latest stable release 2009.2.1 with the patch you have send me. Now the problem is that the last files aren't copied because the device has been umounted automatically.

For example, if i mount the device in '/mnt/test_device' and create the directories 'VideoServer/2009/Mar/20/04' and try to copy the files inside it, when reaching the last files (the ones that couldn't be copied before the patch), the device gets umounted and the copy command fails (because the VideoServer directory doesn't exists because the pendrive has been umounted).

Maybe the patch only works in your development version of the driver and i need to wait for the next stable version.

Thanks for the quick support.


Wed Mar 25, 2009 10:42
Profile
Tuxera CTO

Joined: Tue Nov 21, 2006 23:15
Posts: 1648
Post Re: no space left problem when writing to USB NTFS drive
Hi,

NTFS-3G and nothing else should unmount the volume. I can only image driver crash. Do you have have any sign of this in your log files?

You could enable core file generation by 'ulimit -c 1000000000' then check for the core file in the root directory. If it happens then could you please check with gdb where it crashes: gdb path-to-ntfs-3g-binary /core then type bt at the gdb prompt?

I can't reproduce the crash problem. It's also possible the patch was not correct though I can't see where it could have gone wrong.

Regards, Szaka


Wed Mar 25, 2009 13:14
Profile

Joined: Tue Mar 24, 2009 14:01
Posts: 6
Post Re: no space left problem when writing to USB NTFS drive
Hi again,

I haven't been able to reproduce the problem again. I have correctly applied the patch and the first problem i had has never occurred again. Maybe my first test was wrong and i umounted the unit myself by mistake.

I will continue testing and report if i notice something again.

Thanks very much for the help.


Wed Mar 25, 2009 15:42
Profile
Tuxera CTO

Joined: Tue Nov 21, 2006 23:15
Posts: 1648
Post Re: no space left problem when writing to USB NTFS drive
Hi again,

Can you find anything in your log files related to the unexpected unmount/crash? If the device was unmounted then there must be an

Code:
ntfs-3g[PID_NUMBER]: Unmounting DEVICE ()

entry in your log file. If there was a crash then the kernel should have also log it in /var/log/messages (messages.log, daemnon.log, etc depending on the distribution).

Thanks, Szaka


Wed Mar 25, 2009 15:52
Profile

Joined: Tue Mar 24, 2009 14:01
Posts: 6
Post Re: no space left problem when writing to USB NTFS drive
szaka wrote:
Hi again,

Can you find anything in your log files related to the unexpected unmount/crash? If the device was unmounted then there must be an

Code:
ntfs-3g[PID_NUMBER]: Unmounting DEVICE ()

entry in your log file. If there was a crash then the kernel should have also log it in /var/log/messages (messages.log, daemnon.log, etc depending on the distribution).

Thanks, Szaka


I haven't found anything related to a crash or an error in my log files. No unexpected umount line have been logged. In the following tests i have completed everything worked fine.

I will continue testing and report if found something.

Regards, Antonio.


Wed Mar 25, 2009 15:56
Profile
Tuxera CTO

Joined: Tue Nov 21, 2006 23:15
Posts: 1648
Post Re: no space left problem when writing to USB NTFS drive
I can't produce any crash.

Some words about the Unmount log entry. Every mount is logged like

Code:
ntfs-3g [PID] : Mounted /dev/hda3

All of them are coupled with an Unmount log entry as above during normal operation (no power loss, etc).

If one Unmount log entry is missing then it means the driver crashed. If you know the time when you did the copy and you can find the relevant Unmount entry in the log file then the volume was unmounted on demand/request (accident or something triggered it for some reason) and we don't need to investigate and try to reproduce crash.

The driver never unmounts the volume itself. Unmount always happens following an external event , for instance user device eject request, hotplug, etc.

Thanks, Szaka


Wed Mar 25, 2009 16:19
Profile

Joined: Wed Mar 31, 2010 09:33
Posts: 9
Post Re: no space left problem when writing to USB NTFS drive
please allow me jumping in on this topic:

After having problems using extended (german) characters on the USB-attached NTFS drive mounted with Ubuntu's on-board ntfs, I found your ntfs-3g product last week.
So far, so good, the character handling does work as promised.
But there occurs another problem: this USB drive is also shared as samba resource for some windows computer. And here it comes: trying to copy files via filemanager or syncback to it with "no space left" - but I can create new files or directories using filemanager ...
And of course I can copy or rsync these same files to the USB disk from an internal ext3 drive (which I shared via samba to get the windows files onto).

The system is running Ubuntu 8.04 LTS (2.6.28-11-server #42-Ubuntu SMP x86_64)
ntfs-3g is 2010-03-06 (so including the afore mentioned patch?)
The mount comes implicitly from /etc/fstab:
/dev/sdd1 /media/usbdisk ntfs-3g default,user,sync,exec 0 0
the drive is a 500GB with only less than 5% free space left

appreciating any help
Michael


Wed Mar 31, 2010 10:03
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: no space left problem when writing to USB NTFS drive
Hi,

Quote:
And here it comes: trying to copy files via filemanager or syncback to it with "no space left" - but I can create new files or directories using filemanager ...

Is the error showing on a Windows client screen ?
Quote:
ntfs-3g is 2010-03-06 (so including the afore mentioned patch?)

Which patch are you mentioning (ntfs-3g-2010.3.6 is the latest stable version) ?
Code:
/dev/sdd1 /media/usbdisk ntfs-3g default,user,sync,exec 0 0

That is correct, though sync is unusual.
Quote:
the drive is a 500GB with only less than 5% free space left

Is 5% what "df /media/usbdisk" returns ?
If the error is shown on a Windows screen, this is caused by Windows checking the space left before copying. For instance if you have 8MB free space and you want to copy a 10MB file into a directory marked for compression, Windows will complains, even though 5MB may be actually needed.
What is the free space shown from the Windows point of view ?

Regards

Jean-Pierre


Wed Mar 31, 2010 12:30
Profile

Joined: Wed Mar 31, 2010 09:33
Posts: 9
Post Re: no space left problem when writing to USB NTFS drive
Salut,

jpa wrote:
Is the error showing on a Windows client screen ?

Which patch are you mentioning (ntfs-3g-2010.3.6 is the latest stable version) ?

Is 5% what "df /media/usbdisk" returns ?

What is the free space shown from the Windows point of view ?


yes, this comes on the windows screen, as well in file manager as in CMD-box.

within this topic, about one year ago szaka did mention a patch for the then actual version.

yes, the 5% are returned by df - and on a 500GB disk, this means still 25GB, which should be really enough for copying even 50 files of 240kB ... and I ran into the problem on wanting to copy 5 of them ...
Windows file manager tells me, after subst'ing the UNC, of 17.9GB free of 465GB total.
I can copy boot.ini from the c:\ directory (less than 1 kB), and I can also create new files and directories, but a randomly chosen .inf file of about 5 kB is refused ...

The "sync" parameter I chose to mostly guarantee the coherency of the data, because I must be prepared that the USB disk is unplugged/changed without previous umount'ing - the user does act as on his Windows PC

---

Seems I found something:
Until now, I tried to copy into some specific -very big- directory on the USB disk;
just (by error) copied into some other dir, and it was accepted ...
Are there maybe some constraints regarding file and/or subdirectory numbers?

thanks in advance
Michael


Wed Mar 31, 2010 20:40
Profile

Joined: Wed Mar 31, 2010 09:33
Posts: 9
Post Re: no space left problem when writing to USB NTFS drive
miko wrote:
Are there maybe some constraints regarding file and/or subdirectory numbers?


PS: the directory in question contains 80.861 files in 3.288 directories, totalling to nearly 200 GBytes


Wed Mar 31, 2010 20:46
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: no space left problem when writing to USB NTFS drive
Hi,

Quote:
within this topic, about one year ago szaka did mention a patch for the then actual version.

Ok. This patch is included in all versions since then.
Quote:
Until now, I tried to copy into some specific -very big- directory on the USB disk;
just (by error) copied into some other dir, and it was accepted ...

Access times are higher on a near-full disk and on big directories. This could cause some time-out trigger in Samba.
Quote:
PS: the directory in question contains 80.861 files in 3.288 directories, totalling to nearly 200 GBytes

ntfs-3g has been successfully tested with much more files in a single directory, but Samba can have some limitations on its own. There are some limits in ntfs (for example no more than 30 directory levels), but they are the same for local access and remote access through Samba.

What is the history of this drive ? In what circumstances did the error occur the first time ?
What is the exact number of files and directories at a single level (not counting files in subdirectories) ?

Can you plug the drive into Windows and start a chksdsk ?

Regards

Jean-Pierre


Fri Apr 02, 2010 08:51
Profile

Joined: Wed Mar 31, 2010 09:33
Posts: 9
Post Re: no space left problem when writing to USB NTFS drive
Salut,

jpa wrote:
Ok. This patch is included in all versions since then.

that's what I thought, on first reading the topic i did not regard the dates ...

jpa wrote:
Access times are higher on a near-full disk and on big directories. This could cause some time-out trigger in Samba.

ntfs-3g has been successfully tested with much more files in a single directory, but Samba can have some limitations on its own. There are some limits in ntfs (for example no more than 30 directory levels), but they are the same for local access and remote access through Samba.

Maybe - but this same disk never brought up these problems for over a year, using the "standard" Ubuntu ntfs driver. The samba system was not touched.
Then last week, I came on another, similarly configured machine into the problem with the extended characters, and switched them both into the new ntfs-3g driver, just to keep them both as similar as possible ...

jpa wrote:
What is the history of this drive ? In what circumstances did the error occur the first time ?
What is the exact number of files and directories at a single level (not counting files in subdirectories) ?

The message comes, when trying to copy new files onto the disk into a destination anywhere within the largest top-level directory structure. It does not occur, when I manually create new files or directories within the same directory via Samba, or when I copy very small files to there, or not even when I copy the files into other locations on the drive.

It depends really on the file size:
I just generated an empty txt file, randomly entered a few lines of characters and saved this file.
This small file of -afterwards counted 496 Bytes- I can copy as often as I want.
But when I try to append 5 more characters, the saving is deferred as "not enough space".
So this really seems to be caused by the number of clusters/blocks used on the disk ...
These not-syncback'ed files I can Linux-cp into this same location.
So it really comes back into a Samba topic (which I agree to not be your problem) ... that came to light after changing the ntfs driver (about which I'd like to get helpful hints from you).

For better explanation of the situation:
The drive is used as backup media.
It is attached to the mentioned Ubuntu machine, which carries a Windows VM. This virtualized Windows was converted from a previously stand-alone file server, that died due to hardware failure (much simpler than to reinstall everything from scratch ...).
It does its backup in two stages using 2BrightSparks's SyncBack: 1st level onto an internal ext3 drive every 2 hours, 2nd level once a day to that USB drive.
But as I found, the "disk full" message comes up also on cmd's copy or xcopy ...
I want this drive as NTFS to be able to attach it in case of server's failure easily to any user's Windows PC for quick data access.
jpa wrote:
Can you plug the drive into Windows and start a chksdsk ?

That would be possible, but only next week, after the holidays - remote acces is not capable of everything ;) ...

Thanks for reading
Michael


Fri Apr 02, 2010 11:02
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: no space left problem when writing to USB NTFS drive
Hi,

Quote:
Maybe - but this same disk never brought up these problems for over a year, using the "standard" Ubuntu ntfs driver.

Do you know what ntfs-3g version was this previous one ?
Quote:
The message comes, when trying to copy new files onto the disk into a destination anywhere within the largest top-level directory structure.

Can you remount your device with debug option, trigger an error (try to be minimal), and post the debug information (please gzip it and attach) ?
Code:
# as root :
umount  /media/usbdisk
ntfs-3g -o debug,sync /dev/sdd1 /media/usbdisk 2> debug.log
# trigger an error condition
umount  /media/usbdisk
gzip debug.log


Regards

Jean-Pierre


Fri Apr 02, 2010 12:27
Profile

Joined: Wed Mar 31, 2010 09:33
Posts: 9
Post Re: no space left problem when writing to USB NTFS drive
Salut again,

thanks for the quick reply.

I did as advised - here#s the debug.log.

Although it was not so easy to umount afterwards - the disk stays busy after accessing it vie file manager from windows.
Had to restart Samba to get it free ...

thanks
Michael


Attachments:
debug.log.gz [45.03 KiB]
Downloaded 838 times
Fri Apr 02, 2010 12:53
Profile

Joined: Wed Mar 31, 2010 09:33
Posts: 9
Post Re: no space left problem when writing to USB NTFS drive
oh, sorry, forgot something:

the previously used ntfs was the standard version fom Ubuntu8.04, should've been 1.22something ...


Fri Apr 02, 2010 12:54
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: no space left problem when writing to USB NTFS drive
Hi,

I suppose the file you were trying to create was "/server-d/DATEN/Saarxxxxxxxx.DOC", which you were feeding from a file with same name within another directory. I see the first write() was rejected with error code "Operation not supported". I suspect you have reached some internal limit which I have to dig more to explain.

Can you confirm this is the right file, before I dig ?

Also, has something been written to the system log (generally in /var/log/*) during this debug mounting ?

*edit*

Also : how many files in server-d/DATEN ?
Code:
ls -ldi <mountpoint>/server-d/DATEN
ls -l <mountpoint>/server-d/DATEN | wc


Regards

Jean-Pierre


Fri Apr 02, 2010 14:23
Profile

Joined: Wed Mar 31, 2010 09:33
Posts: 9
Post Re: no space left problem when writing to USB NTFS drive
hey, that's quick ...

yes, that is the file

ls -ldi:
46204 drwxrwxrws 1 root root 65536 2010-04-02 12:47 /media/usbdisk/server-d/DATEN

ls -l ... | wc:
356 3090 22513


I really appreciate your help
Michael


Fri Apr 02, 2010 19:16
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: no space left problem when writing to USB NTFS drive
Hi Michael,

I have examined all the situations in which ntfs-3g returns "not supported", and there is a single one compatible with your debug data : this is when you try to overwrite an existing compressed file (only appending and clearing are implemented).

When copying a file through Samba, you possibly do the following (not sure about it, but this is compatible with the data) :
- create the file
- truncate the file to target size
- write from the beginning
In that sequence, the write is an overwrite, which is not implemented if the file is compressed.

This explains why you can copy locally, and why you can copy through Samba into another directory. It also explains why you can write small files : compression only takes place when the data does not fit into the inode.

To confirm this, please display and post the attribs of the parent directory :
Code:
getfattr -e hex -n system.ntfs_attrib /media/usbdisk/server-d/DATEN

The obvious solution is to remove the compression flag :
Code:
setfattr -v 0x10000000 -n system.ntfs_attrib  /media/usbdisk/server-d/DATEN


Now, your disk is nearly full, and compression was probably intended to save space, so you might also try other ways for copying, so that the file is not forced to target size from the beginning (eg something equivalent to "cat sourcefile > targetfile"). You may also copy to a temporary directory, then copy locally - not move ! - into the compressed directory.

Regards

Jean-Pierre


Sat Apr 03, 2010 08:40
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 28 posts ]  Go to page 1, 2  Next


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.