FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Sat May 15, 2021 04:42



Post new topic Reply to topic  [ 79 posts ]  Go to page Previous  1, 2, 3, 4  Next
Operation not supported (45) - ACL problem 
Author Message

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
Hi Jean-Pierre. Sorry it's taken a while to respond.

jpa wrote:
And did you try mounting your ext3 partition with option noacl ?

I have not been able to try this yet. I'm a bit worried about messing around too much with the device since my sister in now using this device in the UK to store files for her business. She can't really afford the machine to not be working for any length of time. When I tried the "nt acl support = no" test it completely locked her out from the NAS for a few hours until I could log in from New York to fix it the next morning.

jpa wrote:
By the way, can you make an ACL copy from ext3 to ext3 ?

Yes, that works fine.
Code:
getfacl INVOICE.doc
# file: INVOICE.doc
# owner: karen
# group: users
user::rw-
user:lifepractice:rw-
user:mshields:rw-
group::rw-
mask::rwx
other::---

cp -p INVOICE.doc  test.doc
getfacl test.doc
# file: test.doc
# owner: karen
# group: users
user::rw-
user:lifepractice:rw-
user:mshields:rw-
group::rw-
mask::rwx
other::---



Also please confirm that "ls -l" does not show the presence of an ACL on ext3, just to locate where the ACLs are dropped.
Code:
ls -l test.doc
-rw-rwx---    1 karen    users      126464 Oct  8 13:21 test.doc


jpa wrote:
Yes, I can build a configuration which works... but I will still have to make sure that this is what the users want... and if I understand correctly that would not be what you need !

My view on what users want is to be able to copy files to/from various source file systems (EXT3, NTFS, FAT32) from/to NTFS with as little permissions loss as possible (within the constraints of what is feasible within each file system and operating system).

When a Windows user that is logged into a given domain creates a file on a remote Samba share from one PC and then logs into a different PC on the same domain and attempts to access that same file, then he should see identical permissions from either of the two PCs. Ideally even if the file is on a detachable USB NTFS harddrive, it should be possible to take that drive and connect it to any PC in the domain and also see that the correct intended file ownership remains. The ability to query all the possible SIDs for a given user across all the PC's in the domain and set permissions on the NTFS drive to allow consistent permissions for all those SIDs seems like a useful feature if you can implement it. But as you say, it is probably not a feature that I need at the moment (but other users certainly will).

In the case where PC's are not part of a domain, then I'm not sure you will be able to get the SIDs. In this case, I guess you just have to do your best to lose as little permission info as possible in the three cases of:
1) Copying from EXT3 to NTFS in a Linux machine (like my Readynas)
2) Copying from NTFS back to EXT3 in a Linux machine
3) Connecting the NTFS drive to a Windows PC and trying to read/write to it

I'm still not sure I understand why ACL support is a compile-time option rather than a run-time option. It would be more user-friendly in my view to be a run-time option. If a company like Netgear just compiles your standard build then it's likely that ACL support will be excluded and then it will be almost impossible for users to switch on ACL support without hacking into the machine and voiding their warranty.

jpa wrote:
Also, what processor are you using ? (if big endian, I may have another explanation).

The readynas uses some kind of custom Sparc processor. I'm not sure of the endian-ness but I would guess it's big-endian.

jpa wrote:
Before doing the above test, can you make a simple check to make sure the ACL settings reach ntfs-3g. Simply try to do a setfacl on a NOT OWNED file. This must of course NOT be done as root

It seems that I can only run as root on this machine and it won't let me switch user (see what happens when I try below). Can you suggest any test that I could perform as root?
Code:
tera:~# whoami
root
tera:~# su - mark
tera:~# whoami
root


jpa wrote:
At the moment I can only imagine some data alignment problem, and if is a big-endian one, please test the special version suggested before.

It's not very easy for me to compile new versions of NTFS-3G since I don't have a development environment to build it correctly. With the previous version that you suggested I try I got the help of a developer on the Netgear website (he has a full development environment with gcc etc set up on an old Sun Sparc machine). He might be persuaded to help again. But before I contact him and go to this trouble, is there any other troubleshooting or diagnosis that we can try? Is it possible perhaps to switch on some logging (at runtime) in NTFS-3G so that we can see where the error is occurring?


Fri Oct 15, 2010 23:34
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi,

Quote:
My view on what users want is to be able to copy files to/from various source file systems (EXT3, NTFS, FAT32) from/to NTFS with as little permissions loss as possible (within the constraints of what is feasible within each file system and operating system).

I agree, that is also what I aim at (for professionnal environments), but most users (home users) do not want any permission checks at all. The default options are best for users who do not want to define a permissions strategy.
Quote:
When a Windows user that is logged into a given domain creates a file on a remote Samba share from one PC and then logs into a different PC on the same domain and attempts to access that same file, then he should see identical permissions from either of the two PCs. Ideally even if the file is on a detachable USB NTFS harddrive, it should be possible to take that drive and connect it to any PC in the domain and also see that the correct intended file ownership remains. The ability to query all the possible SIDs for a given user across all the PC's in the domain and set permissions on the NTFS drive to allow consistent permissions for all those SIDs seems like a useful feature if you can implement it. But as you say, it is probably not a feature that I need at the moment (but other users certainly will).

I am waiting for somebody using such configuration to define and test a proper strategy. I have already posted a script to query the unified SIDs across a domain, but I got no answer.
Quote:
In the case where PC's are not part of a domain, then I'm not sure you will be able to get the SIDs.

Well, in that situation, according to your previous posts, there is no unified SID, and a user with the same login on different PCs have a different SID on each one. So a file created one one machine appears as owned by a foreign user on a different machine though having the same login name. A file cannot have several owners, I have to choose, and either use a predefined policy or be given the SIDs to use.

Inputs in this area much welcome.

Quote:
In this case, I guess you just have to do your best to lose as little permission info as possible in the three cases of:
1) Copying from EXT3 to NTFS in a Linux machine (like my Readynas)
2) Copying from NTFS back to EXT3 in a Linux machine
3) Connecting the NTFS drive to a Windows PC and trying to read/write to it

All these are done. However in the third situation, you have to predefine the Windows PC with which there is full compatibility. In most cases the files are created as readable by anybody, but your previous posts showed files which could only be read by same owner or same group (could be because your umask was set to 0770, affecting your ext3 files).
Quote:
I'm still not sure I understand why ACL support is a compile-time option rather than a run-time option.

Maybe i will do it some time if there is enough demand. It will require rewriting and I will have to deal with differences among OS'es as Posix ACLs only exist on Linux and I am uneasy to get configurations right for OS'es I do not have access to.
Quote:
If a company like Netgear just compiles your standard build then it's likely that ACL support will be excluded and then it will be almost impossible for users to switch on ACL support without hacking into the machine and voiding their warranty.

The "standard build" is intended for standard usage, but Netgear probably promotes more advanced features which could be reflected in their specific builds...
Quote:
The readynas uses some kind of custom Sparc processor. I'm not sure of the endian-ness but I would guess it's big-endian.

This is an important point. can you try :
Code:
getfattr -e hex -n system.ntfs_times <some ntfs file>

This will return the time stamps of the file, represented according to the endiannes of the processor.
Code:
# on a small endian processor, the result is likely to end with cb01 :
system.ntfs_times=0xca52e8388056cb015829ccfb036dcb015829ccfb036dcb015829ccfb036dcb01
# on a big endian processor, the result is likely to begin with 01cb :
system.ntfs_times=0x01cb568038e852ca01cb6d03fbcc295801cb6d03fbcc295801cb6d03fbcc2958

I think the recent Sparcs can be used both ways. If it is used as small endian, you need not try the test version I had suggested. I have rechecked the ACL library source, and it appears that it uses a small endian interface even on big endian computers, which is not what ntfs-3g expected, hence the proposed fix. Better be sure about the endianness before trying the test version.
Quote:
It seems that I can only run as root on this machine and it won't let me switch user (see what happens when I try below). Can you suggest any test that I could perform as root?

I could not find one, because most of the checks are done by the library before the requests reach ntfs-3g, and virtually anything is allowed to root.
Quote:
It's not very easy for me to compile new versions of NTFS-3G since I don't have a development environment to build it correctly.

That is understandable, and most users would not be able to do the same. Thank you for your tests and sharing your experience.

Let us first clarify the endianness issue.

Regards

Jean-Pierre


Sat Oct 16, 2010 12:28
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi again,

Sorry, I forgot :
Quote:
Is it possible perhaps to switch on some logging (at runtime) in NTFS-3G so that we can see where the error is occurring?

Yes, the following may be useful : mount with debug option, do a chmod on some file (just for identifying the file), then do a getfacl (so that the log shows it reaches fuse), then unmount (nothing else, to avoid ambiguities in the interpretation) :
Code:
ntfs-3g -o debug <device> <mountpoint> 2> logfile
chmod 444 <some file>
getfacl <same file>
umount <mountpoint>

In the log, you should find a single SETATTR corresponding the the chmod, and a GETXATTR with the same nodeid :
Code:
unique: 386, opcode: SETATTR (4), nodeid: 16, insize: 128
   unique: 386, error: 0 (Success), outsize: 120
[...]
unique: 390, opcode: GETXATTR (22), nodeid: 16, insize: 72
   unique: 390, error: 0 (Success), outsize: 68

If there is a GETXATTR, ntfs-3g is called and the Posix ACL library is very likely to be implemented (this means a probable bug in ntfs-3g).

Regards

Jean-Pierre


Sat Oct 16, 2010 12:49
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
jpa wrote:
Quote:
My view on what users want is to be able to copy files to/from various source file systems (EXT3, NTFS, FAT32) from/to NTFS with as little permissions loss as possible (within the constraints of what is feasible within each file system and operating system).

I agree, that is also what I aim at (for professionnal environments), but most users (home users) do not want any permission checks at all. The default options are best for users who do not want to define a permissions strategy.

I don't know if this is even possible, but in my situation (user mode not domain mode) I would ideally like to be able to define a Linux permissions strategy, but perhaps not define a Windows permissions strategy. What I mean by that is I would like any user to be able to disconnect the USB drive from my NAS and attach to any windows PC and have full access to all the files on the drive. At the same time, I would want the Linux permissions to be retained so that if I had some disaster and all the data on my NAS were lost, I could simply copy (cp -p) all the files back from the USB drive onto a new empty ext3 NAS and recover all my data with ownership & permissions intact as well (assuming I am careful that the Linux UID and GID numbers on the USB drive match with the correct UIDs (/etc/passwd) and GIDs (/etc/group) set up on the NAS. I don't really care so much about retaining the ACLs, but if it is also possible to retain ACLs as well as the Linux ownership & permissions when restoring to Linux ext3 from a NTFS USB backup drive that would be nice too. Is there a way to achieve this with NTFS-3G?

My current situation is this: Every single file that has been copied (cp -p) from ext3 to the USB NTFS drive has permissions 777 and ownership root/root
Code:
ls -l
total 305
-rwxrwxrwx    1 root     root        37376 Jul 10  2009 client record card.doc
-rwxrwxrwx    1 root     root        31232 Jul 11  2009 history.doc
-rwxrwxrwx    1 root     root       126464 Oct  8 13:21 INVOICE.doc
-rwxrwxrwx    1 root     root        57856 Sep  1 16:49 session notes.doc

So Linux permission info has been lost despite the use of cp -p. And getfacl shows a similar picture
Code:
getfacl INVOICE.doc
# file: INVOICE.doc
# owner: root
# group: root
user::rwx
group::rwx
other::rwx

So if I ever need to use this backup to restore to ext3 I will have to manually recover the ownership & permissions myself somehow since the permissions have not been stored to NTFS. My primary concern is not to secure the data on the drive (if I wanted that then permissions probably wouldn't help - I would need to encrypt the drive), but rather to be able to use it as an effective backup/restore solution in a mixed Windows/Linux environment.
In the event of the NAS failing completely, the users might not be able to wait for the NAS to be fixed. So I need a way for them to be able to access files directly from the USB NTFS backup in an emergency (at a minimum any PC user should have read-access to all the files). At some point it is likely that the NAS will be repaired/replaced and then we will want to restore the data from the USB NTFS drive back onto the NAS (now I would want to restore not only the files but also the ownership & permissions - I don't want every file to be root/root 777).
jpa wrote:
Well, in that situation, according to your previous posts, there is no unified SID, and a user with the same login on different PCs have a different SID on each one. So a file created one one machine appears as owned by a foreign user on a different machine though having the same login name. A file cannot have several owners, I have to choose, and either use a predefined policy or be given the SIDs to use.

Inputs in this area much welcome.

Idea 1
Perhaps you have already considered this, but I suppose that you could allow users to define a (1 to N) mapping from Linux user/UID to windows SID (it is a one to N mapping when there is more than one Windows SID that should be considered as an alias for the same Linux user). This way when Linux file ownership is changed for a file (e.g chown), you could perhaps ensure that all the mapped SIDs have the appropriate ACL permissions for the file. This doesn't solve the problem of having to pick one to be the owner, but it should at least allow correct accessibility to the files. But then it gets tricky - what if I buy a new PC, now I have another Windows SID for Linux user "mark". I would need a tool to go through all the files on the NTFS drive and add this new SID to all the existing files owned by "mark". It would probably be most convenient if this tool were also available on Windows (since at the point a user needs this tool, their Linux NAS may not be functioning).

Idea 2
Perhaps there's another approach. Thinking about this problem a bit more, this lack of common SID isn't a problem that is limited to Windows. I've seen it on Linux and Unix too when a file is created on one system, but then read in another system if the UIDs and GIDs are not synchronized across both machines (either manually or through using a service like NIS). When there is no NIS service to synchonize UIDs and GIDs my solution in the past has been to manually change the UIDs and GIDs to be consistent on all of the machines that I care about. Perhaps the same could be done on Windows. I found a tool called "NewSID" that can change the SID for a Windows user (http://technet.microsoft.com/en-us/sysi ... 97418.aspx). Unfortunately it sounds like it was retired in 2006 for reasons stated in this article (http://blogs.technet.com/b/markrussinov ... 91024.aspx). Interestingly the last article comes to the conclusion that there is no harm to have duplicate SIDs in the same network. They also say
Quote:
Some articles on SID duplication, including this KB article, warn that if multiple computers have the same SID, that resources on removable media like an NTFS-formatted firewire disk can’t be secured to a local account.

This is not necessarily a bad thing in my view. It sounds to me like SID duplication gives us exactly the behaviour that we want - allow the files to be accessed by the same user "mark" on any of multiple PCs. It would also solve the problem of file ownership - now there would only be one SID per unique user name.

Idea 3
There is one more approach I can think of. Is there perhaps a way (without losing the Linux permissions information) to make it so that when the USB NTFS drive is disconnected from the NAS and instead connected to any Windows PC, all Windows users have full access to every file? Ultimately this would be the simplest approach from my perspective (requiring no additional SID mapping or synchronization at all). The key point here though is that we don't want to lose the Linux file ownership and permissions if possible.

jpa wrote:
Maybe i will do it some time if there is enough demand. It will require rewriting and I will have to deal with differences among OS'es as Posix ACLs only exist on Linux and I am uneasy to get configurations right for OS'es I do not have access to.

Would it be a quicker fix if you simply made the default to have ACLs on ONLY for Linux and off for any other OS? What other OS's does NTFS-3G build on currently?
jpa wrote:
The "standard build" is intended for standard usage, but Netgear probably promotes more advanced features which could be reflected in their specific builds...

Well, as far as I know the ACL option is off in all the Netgear Readynas devices. Perhaps it was an intentional decision but I think it's more likely that Netgear would just build all packages (not just NTFS-3G) using the defaults and if it passes their tests, then release that configuration to end-users rather than risking any non-standard build options that they don't have time to fully investigate, understand and then document for users.

It seems that the machine is definitely big-endian
Code:
getfattr -e hex -n system.ntfs_times INVOICE.doc
# file: INVOICE.doc
system.ntfs_times=0x01cb673d577d703001cb66e35e8b7c0001cb673d56f8c10001cb673d5789add2


I also confirmed this via shell script using
Code:
On a Big Endian-System (Solaris)

> echo -n I | od -to2 | head -n1 | cut -f2 -d" " | cut -c6
0

On a little endian system (Linux, Intel)

$ echo -n I | od -to2 | head -n1 | cut -f2 -d" " | cut -c6
1


I will next try to create a log file as per your instructions.


Mon Oct 18, 2010 22:26
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
So I executed the following commands
Code:
ntfs-3g -o debug,rw,sync,noatime,allow_other,blksize=4096 /dev/sda1 /tmpusb 2> /tmp/ntfg-3g.err
cd /tmpusb/
ls
cd marktest
ls
chmod 444 test
getfacl test
umount /dev/sda1


Here is what I get:
Code:
cat /tmp/ntfg-3g.err
WARNING: blksize option is ignored because ntfs-3g must calculate it.
utime_omit_ok: 0
Version 2010.8.8 integrated FUSE 28
Mounted /dev/sda1 (Read-Write, label "OneTouch4", NTFS 3.1)
Cmdline options: debug,rw,sync,noatime,allow_other,blksize=4096
Mount options: rw,sync,allow_other,allow_other,nonempty,noatime,fsname=/dev/sda1,blkdev,blksize=4096
Ownership and permissions disabled, configuration type 7
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.8
flags=0x00000003
max_readahead=0x00020000
   INIT: 7.8
   flags=0x00000041
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, error: 0 (Success), outsize: 40
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 40
   unique: 2, error: 0 (Success), outsize: 112
unique: 3, opcode: GETATTR (3), nodeid: 1, insize: 40
   unique: 3, error: 0 (Success), outsize: 112
unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 40
   unique: 4, error: 0 (Success), outsize: 112
unique: 5, opcode: OPENDIR (27), nodeid: 1, insize: 48
   unique: 5, error: 0 (Success), outsize: 32
unique: 6, opcode: GETATTR (3), nodeid: 1, insize: 40
   unique: 6, error: 0 (Success), outsize: 112
unique: 7, opcode: READDIR (28), nodeid: 1, insize: 64
   unique: 7, error: 0 (Success), outsize: 688
unique: 8, opcode: READDIR (28), nodeid: 1, insize: 64
   unique: 8, error: 0 (Success), outsize: 16
unique: 9, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
   unique: 9, error: 0 (Success), outsize: 16
unique: 10, opcode: GETATTR (3), nodeid: 1, insize: 40
   unique: 10, error: 0 (Success), outsize: 112
unique: 11, opcode: OPENDIR (27), nodeid: 1, insize: 48
   unique: 11, error: 0 (Success), outsize: 32
unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 40
   unique: 12, error: 0 (Success), outsize: 112
unique: 13, opcode: READDIR (28), nodeid: 1, insize: 64
   unique: 13, error: 0 (Success), outsize: 688
unique: 14, opcode: READDIR (28), nodeid: 1, insize: 64
   unique: 14, error: 0 (Success), outsize: 16
unique: 15, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
   unique: 15, error: 0 (Success), outsize: 16
unique: 16, opcode: LOOKUP (1), nodeid: 1, insize: 49
LOOKUP /marktest
   NODEID: 2
   unique: 16, error: 0 (Success), outsize: 136
unique: 17, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 17, error: 0 (Success), outsize: 112
unique: 18, opcode: GETATTR (3), nodeid: 1, insize: 40
   unique: 18, error: 0 (Success), outsize: 112
unique: 19, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 19, error: 0 (Success), outsize: 112
unique: 20, opcode: OPENDIR (27), nodeid: 2, insize: 48
   unique: 20, error: 0 (Success), outsize: 32
unique: 21, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 21, error: 0 (Success), outsize: 112
unique: 22, opcode: READDIR (28), nodeid: 2, insize: 64
   unique: 22, error: 0 (Success), outsize: 312
unique: 23, opcode: READDIR (28), nodeid: 2, insize: 64
   unique: 23, error: 0 (Success), outsize: 16
unique: 24, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
   unique: 24, error: 0 (Success), outsize: 16
unique: 25, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 25, error: 0 (Success), outsize: 112
unique: 26, opcode: LOOKUP (1), nodeid: 2, insize: 45
LOOKUP /marktest/test
   NODEID: 3
   unique: 26, error: 0 (Success), outsize: 136
unique: 27, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 27, error: 0 (Success), outsize: 112
unique: 28, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 28, error: 0 (Success), outsize: 112
unique: 29, opcode: LOOKUP (1), nodeid: 2, insize: 45
LOOKUP /marktest/test
   NODEID: 3
   unique: 29, error: 0 (Success), outsize: 136
unique: 30, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 30, error: 0 (Success), outsize: 112
unique: 31, opcode: GETXATTR (22), nodeid: 3, insize: 72
   unique: 31, error: -45 (Operation not supported), outsize: 16
unique: 32, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 32, error: 0 (Success), outsize: 112
unique: 33, opcode: DESTROY (38), nodeid: 0, insize: 40
Unmounting /dev/sda1 (OneTouch4)
   unique: 33, error: 0 (Success), outsize: 16


So it looks like I am seeing the GETXATTR (22), but rather than getting a "Success" I am instead getting "error: -45 (Operation not supported)".

Regards

Mark


Mon Oct 18, 2010 23:02
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi Mark,

Quote:
It seems that the machine is definitely big-endian

This is an important point. As I said previously, I have recently been aware that the ACL library nevertheless submits small-endian ACLs to the driver. This is not what ntfs-3g expected, and which has been fixed in the test version I suggested, and is also fixed in the advanced version ntfs-3g-2010.10.2AR.1 which I have just released on http://www.tuxera.com/community/ntfs-3g-advanced/ (but not fixed in the standard version ntfs-3g-2010.10.2)
Code:
Version 2010.8.8 integrated FUSE 28
Mounted /dev/sda1 (Read-Write, label "OneTouch4", NTFS 3.1)
Cmdline options: debug,rw,sync,noatime,allow_other,blksize=4096

Note that the sync option was not implemented in ntfs-3g-2010.8.8, it has just been implemented and it is available in ntfs-3g-2010.10.2AR.1 This may be useful to you as you are using kernel 2.6.17 on which umount does not wait for the device to be synced.
Code:
Ownership and permissions disabled, configuration type 7
[...]
unique: 31, opcode: GETXATTR (22), nodeid: 3, insize: 72
   unique: 31, error: -45 (Operation not supported), outsize: 16

The reason why you get "Operation not supported" is you have not defined in .NTFS-3G/UserMapping the SIDs to be used, so no protection can be enabled.
However the important result here is that the GETXATTR is logged, which proves ntfs-3g is called, so checking the proposed fix is relevant. We however see that the error is not returned to the application, another hitch in this early stage of integration of ACLs into fuse.

I will reply later to your other detailed remarks.

Regards

Jean-Pierre


Tue Oct 19, 2010 09:43
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
I'm not sure if this is useful, but here's some additional information. Here is my /etc/fstab file.
Code:
cat /etc/fstab
#-----------------------------------------------------------------------------
# <device>         <mount>              <type>     <options>    <freq> <pass>
#-----------------------------------------------------------------------------
/dev/hdc1          /                    ext3       defaults,noatime      0      0
proc               /proc                proc       defaults          0      0
/dev/hdc2          swap                 swap       defaults          0      0
/dev/hdg2          swap                 swap       defaults          0      0
/dev/c/c           /c                   ext3       defaults,acl,user_xattr,usrquota,grpquota,noatime,data=journal      0      2
/cam               /home/ftp/cam        bind       bind              0      0
/lifepractice      /home/ftp/lifepractice bind       bind              0      0


And here is the output from mount:
Code:
mount
/dev/hdc1 on / type ext3 (rw,noatime)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw)
sysfs on /sys type sysfs (rw)
tmpfs on /ramfs type ramfs (rw)
tmpfs on /USB type tmpfs (rw,size=16k)
/dev/c/c on /c type ext3 (rw,noatime,acl,user_xattr,usrquota,grpquota,data=journal)
/c/cam on /home/ftp/cam type bind (rw,bind)
/c/lifepractice on /home/ftp/lifepractice type bind (rw,bind)
nfsd on /proc/fs/nfsd type nfsd (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /tmpusb type fuseblk (rw,sync,noatime,allow_other,allow_other,blksize=4096)


There are a couple of things that seem strange to me.
1) Why is it that /dev/sda1 does not have an entry in /etc/fstab? Does NTFS-3G not use fstab?
2) The entry for /dev/sda1 from mount does not contain the options acl,user_xattr even though I explicitly specified both options in the mount command as follows:
Code:
ntfs-3g -o debug,acl,user_xattr,rw,sync,noatime,allow_other,blksize=4096 /dev/sda1 /tmpusb 2> /tmp/ntfg-3g.err

I noticed that the filesystem mounted at / (/dev/hdc1) also does not specify acl,user_xattr. So I tried to copy a file containing ACLs to /tmp and it fails too
Code:
cp -p INVOICE.doc /tmp
cp: preserving permissions for `/tmp/INVOICE.doc': Operation not supported


When I copy the same file to the NTFS filesystem I get the identical error (which makes me wonder whether the lack of acl,user_xattr from mount in both cases may be a factor here) :
Code:
cp -p INVOICE.doc /tmpusb/marktest/
cp: preserving permissions for `/tmpusb/marktest/INVOICE.doc': Operation not supported


And here is the corresponding debug log
Code:
WARNING: blksize option is ignored because ntfs-3g must calculate it.
utime_omit_ok: 0
Version 2010.8.8 integrated FUSE 28
Mounted /dev/sda1 (Read-Write, label "OneTouch4", NTFS 3.1)
Cmdline options: debug,acl,user_xattr,rw,sync,noatime,allow_other,blksize=4096
Mount options: acl,rw,sync,allow_other,allow_other,nonempty,noatime,fsname=/dev/sda1,blkdev,blksize=4096
Ownership and permissions disabled, configuration type 7
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.8
flags=0x00000003
max_readahead=0x00020000
   INIT: 7.8
   flags=0x00000041
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, error: 0 (Success), outsize: 40
tera:~# tail -f /tmp/ntfg-3g.err
Ownership and permissions disabled, configuration type 7
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.8
flags=0x00000003
max_readahead=0x00020000
   INIT: 7.8
   flags=0x00000041
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, error: 0 (Success), outsize: 40
unique: 2, opcode: LOOKUP (1), nodeid: 1, insize: 49
LOOKUP /marktest
   NODEID: 2
   unique: 2, error: 0 (Success), outsize: 136
unique: 3, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 3, error: 0 (Success), outsize: 112
unique: 4, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 4, error: 0 (Success), outsize: 112
unique: 5, opcode: LOOKUP (1), nodeid: 2, insize: 52
LOOKUP /marktest/INVOICE.doc
   NODEID: 3
   unique: 5, error: 0 (Success), outsize: 136
unique: 6, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 6, error: 0 (Success), outsize: 112
unique: 7, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 7, error: 0 (Success), outsize: 112
unique: 8, opcode: OPEN (14), nodeid: 3, insize: 48
   unique: 8, error: 0 (Success), outsize: 32
OPEN[0] flags: 0x40001 /marktest/INVOICE.doc
unique: 9, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 9, error: 0 (Success), outsize: 112
unique: 10, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 0
   WRITE[0] 16384 bytes
   unique: 10, error: 0 (Success), outsize: 24
unique: 11, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 16384
   WRITE[0] 16384 bytes
   unique: 11, error: 0 (Success), outsize: 24
unique: 12, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 32768
   WRITE[0] 16384 bytes
   unique: 12, error: 0 (Success), outsize: 24
unique: 13, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 49152
   WRITE[0] 16384 bytes
   unique: 13, error: 0 (Success), outsize: 24
unique: 14, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 65536
   WRITE[0] 16384 bytes
   unique: 14, error: 0 (Success), outsize: 24
unique: 15, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 81920
   WRITE[0] 16384 bytes
   unique: 15, error: 0 (Success), outsize: 24
unique: 16, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 98304
   WRITE[0] 16384 bytes
   unique: 16, error: 0 (Success), outsize: 24
unique: 17, opcode: WRITE (16), nodeid: 3, insize: 11840
WRITE[0] 11776 bytes to 114688
   WRITE[0] 11776 bytes
   unique: 17, error: 0 (Success), outsize: 24
unique: 18, opcode: FLUSH (25), nodeid: 3, insize: 64
FLUSH[0]
   unique: 18, error: -90 (Function not implemented), outsize: 16
unique: 19, opcode: RELEASE (18), nodeid: 3, insize: 64
RELEASE[0] flags: 0x40001
   unique: 19, error: 0 (Success), outsize: 16
unique: 20, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 20, error: 0 (Success), outsize: 112
unique: 21, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 21, error: 0 (Success), outsize: 112
unique: 22, opcode: SETXATTR (21), nodeid: 3, insize: 124
   unique: 22, error: -45 (Operation not supported), outsize: 16
unique: 23, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 23, error: 0 (Success), outsize: 112


I also repeated the same test adding the option "permissions". The result was identical to the result where I excluded "permissions".

I notice that acl and user_xattr are both present at the top of the debug log, so it seems that NTFS-3G knows that this filesystem supports ACLs, but the mount command (and possibly other areas of the Linux kernel) does not. Could this be the reason that I am getting the "Operation not supported" errors?


Tue Oct 19, 2010 16:57
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
jpa wrote:
The reason why you get "Operation not supported" is you have not defined in .NTFS-3G/UserMapping the SIDs to be used, so no protection can be enabled.

Note that I am getting "Operation not supported" even if I mount using the option "permissions". I think you told me before that if "permissions" is specified in the mount command then .NTFS-3G/UserMapping is not required and NTFS-3G will behave exactly as if "::S-1-5-21-3141592653-589793238-462643383-10000" were placed in this file. Is this correct?

Here is the debug output of the previous test this time with the "permissions" option. Note that I still get "Operation not supported":
Code:
Using default user mapping
utime_omit_ok: 0
Version 2010.8.8 integrated FUSE 28
Mounted /dev/sda1 (Read-Write, label "OneTouch4", NTFS 3.1)
Cmdline options: debug,permissions,acl,user_xattr,rw,sync,noatime,allow_other,blksize=4096
Mount options: acl,rw,sync,allow_other,allow_other,nonempty,default_permissions,noatime,fsname=/dev/sda1,blkdev,blksize=4096
User mapping built, Posix ACLs not used, configuration type 7
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.8
flags=0x00000003
max_readahead=0x00020000
   INIT: 7.8
   flags=0x00000041
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, error: 0 (Success), outsize: 40
unique: 2, opcode: LOOKUP (1), nodeid: 1, insize: 49
LOOKUP /marktest
   NODEID: 2
   unique: 2, error: 0 (Success), outsize: 136
unique: 3, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 3, error: 0 (Success), outsize: 112
unique: 4, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 4, error: 0 (Success), outsize: 112
unique: 5, opcode: LOOKUP (1), nodeid: 2, insize: 52
LOOKUP /marktest/INVOICE.doc
   NODEID: 3
   unique: 5, error: 0 (Success), outsize: 136
unique: 6, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 6, error: 0 (Success), outsize: 112
unique: 7, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 7, error: 0 (Success), outsize: 112
unique: 8, opcode: OPEN (14), nodeid: 3, insize: 48
   unique: 8, error: 0 (Success), outsize: 32
OPEN[0] flags: 0x40001 /marktest/INVOICE.doc
unique: 9, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 9, error: 0 (Success), outsize: 112
unique: 10, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 0
   WRITE[0] 16384 bytes
   unique: 10, error: 0 (Success), outsize: 24
unique: 11, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 16384
   WRITE[0] 16384 bytes
   unique: 11, error: 0 (Success), outsize: 24
unique: 12, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 32768
   WRITE[0] 16384 bytes
   unique: 12, error: 0 (Success), outsize: 24
unique: 13, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 49152
   WRITE[0] 16384 bytes
   unique: 13, error: 0 (Success), outsize: 24
unique: 14, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 65536
   WRITE[0] 16384 bytes
   unique: 14, error: 0 (Success), outsize: 24
unique: 15, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 81920
   WRITE[0] 16384 bytes
   unique: 15, error: 0 (Success), outsize: 24
unique: 16, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 98304
   WRITE[0] 16384 bytes
   unique: 16, error: 0 (Success), outsize: 24
unique: 17, opcode: WRITE (16), nodeid: 3, insize: 11840
WRITE[0] 11776 bytes to 114688
   WRITE[0] 11776 bytes
   unique: 17, error: 0 (Success), outsize: 24
unique: 18, opcode: FLUSH (25), nodeid: 3, insize: 64
FLUSH[0]
   unique: 18, error: -90 (Function not implemented), outsize: 16
unique: 19, opcode: RELEASE (18), nodeid: 3, insize: 64
RELEASE[0] flags: 0x40001
   unique: 19, error: 0 (Success), outsize: 16
unique: 20, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 20, error: 0 (Success), outsize: 112
unique: 21, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 21, error: 0 (Success), outsize: 112
unique: 22, opcode: SETXATTR (21), nodeid: 3, insize: 124
   unique: 22, error: -45 (Operation not supported), outsize: 16
unique: 23, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 23, error: 0 (Success), outsize: 112



Tue Oct 19, 2010 17:04
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
Another interesting thing is that when using the "permissions" option, getfacl now returns something reasonable (it didn't work when we were using the UserMapping file viewtopic.php?p=20437&sid=3593622e2e573ccdfcfcc2757f90033a#p20437):
Code:
# cp -p INVOICE.doc /tmpusb/marktest/
cp: preserving permissions for `/tmpusb/marktest/INVOICE.doc': Operation not supported
# getfacl /tmpusb/marktest/INVOICE.doc
getfacl: Removing leading '/' from absolute path names
# file: tmpusb/marktest/INVOICE.doc
# owner: karen
# group: users
user::rw-
group::rwx
other::---


When I try exactly the same thing with a UserMapping file I get:
Code:
# cp -p INVOICE.doc /tmpusb/marktest/
cp: preserving permissions for `/tmpusb/marktest/INVOICE.doc': Invalid argument

# getfacl /tmpusb/marktest/INVOICE.doc
getfacl: /tmpusb/marktest/INVOICE.doc: Success

[/code]

So it seems to me that the "permissions" option works differently (and seems to work slightly better) than the UserMapping approach.


Tue Oct 19, 2010 17:23
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi again,

[You were posting while I was writing this, I am sending without having read your latest post]

Quote:
I don't know if this is even possible, but in my situation (user mode not domain mode) I would ideally like to be able to define a Linux permissions strategy, but perhaps not define a Windows permissions strategy. What I mean by that is I would like any user to be able to disconnect the USB drive from my NAS and attach to any windows PC and have full access to all the files on the drive.

It must be possible to always add an extra ACL granting full permissions to some group, which would be interpreted by all Windows instances, but not by ntfs-3g. I need to make tries. Of course this would be only done as an option.
Quote:
At the same time, I would want the Linux permissions to be retained so that if I had some disaster and all the data on my NAS were lost, I could simply copy (cp -p) all the files back from the USB drive onto a new empty ext3 NAS and recover all my data with ownership & permissions intact as well (assuming I am careful that the Linux UID and GID numbers on the USB drive match with the correct UIDs (/etc/passwd) and GIDs (/etc/group) set up on the NAS. I don't really care so much about retaining the ACLs, but if it is also possible to retain ACLs as well as the Linux ownership & permissions when restoring to Linux ext3 from a NTFS USB backup drive that would be nice too. Is there a way to achieve this with NTFS-3G?

This is already possible with ntfs-3g (for ownership, permissions and ACLs)
1) either with the default SIDs (the ones you have already used), and having the same numeric uid/gid or editing the UserMapping file before copying to the new NAS,
2) or having the same login/group names, and putting in the UserMapping file the SID to use for each login or group.
Quote:
My current situation is this: Every single file that has been copied (cp -p) from ext3 to the USB NTFS drive has permissions 777 and ownership root/root

This is because you have not defined the user mappings. ntfs-3g does not known which SID to use to record ownership and permissions, and falls back to everything allowed. On Sep 15th you have reported that you could copy ownership and permissions after having defined a standard user mapping (though the ACL issue is not yet solved in your configuration)
Quote:
So if I ever need to use this backup to restore to ext3 I will have to manually recover the ownership & permissions myself somehow since the permissions have not been stored to NTFS. My primary concern is not to secure the data on the drive (if I wanted that then permissions probably wouldn't help - I would need to encrypt the drive), but rather to be able to use it as an effective backup/restore solution in a mixed Windows/Linux environment.

Apart from the ACLs, you can already backup and restore on Linux, and restore on a predefined Windows computer per user (with a Windows tool for also copying the ACLs, but probably not needed if each user can only restore his own files).
Quote:
In the event of the NAS failing completely, the users might not be able to wait for the NAS to be fixed. So I need a way for them to be able to access files directly from the USB NTFS backup in an emergency (at a minimum any PC user should have read-access to all the files).

The main problem is that your files show no permissions for "other" (neither owner nor group). This is what prevents from copying on any Windows computer. Is this intentional in your security policy ?
Quote:
Perhaps you have already considered this, but I suppose that you could allow users to define a (1 to N) mapping from Linux user/UID to windows SID (it is a one to N mapping when there is more than one Windows SID that should be considered as an alias for the same Linux user). This way when Linux file ownership is changed for a file (e.g chown), you could perhaps ensure that all the mapped SIDs have the appropriate ACL permissions for the file. This doesn't solve the problem of having to pick one to be the owner, but it should at least allow correct accessibility to the files.

This is possible (with significant development). It implies merging the permissions for the N instances of the same user to get the permissions seen by Linux.
Quote:
But then it gets tricky - what if I buy a new PC, now I have another Windows SID for Linux user "mark". I would need a tool to go through all the files on the NTFS drive and add this new SID to all the existing files owned by "mark".

Correct. This is a general Windows issue related to the lack of unified SIDs.
Quote:
Perhaps the same could be done on Windows. I found a tool called "NewSID" that can change the SID for a Windows user

Strange article in which the author retires the software because it can cause problems, and he only explains why this cannot be a problem.
Quote:
It sounds to me like SID duplication gives us exactly the behaviour that we want - allow the files to be accessed by the same user "mark" on any of multiple PCs. It would also solve the problem of file ownership - now there would only be one SID per unique user name.

I agree.
Quote:
There is one more approach I can think of. Is there perhaps a way (without losing the Linux permissions information) to make it so that when the USB NTFS drive is disconnected from the NAS and instead connected to any Windows PC, all Windows users have full access to every file?

Maybe this is possible. I have to try.
Quote:
Would it be a quicker fix if you simply made the default to have ACLs on ONLY for Linux and off for any other OS?

Technically I can, but doing so for an integrator who does not want to use a compile option is stupid. And you experience that getting things right even on Linux is not that easy when you cannot test.

I am open to help Netgear to do it and test. That would be the best solution.
Quote:
What other OS's does NTFS-3G build on currently?

MacOSX, FreeBSD, OpenSolaris and OSes for embedded systems I do not know about.

Regards

Jean-Pierre


Tue Oct 19, 2010 17:54
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
After some googling I also found this post (I think you are the person who replied actually):
http://comments.gmane.org/gmane.comp.fi ... devel/8273

The symptoms of the poster's problem sound very similar to my problem. I wonder whether the "Operation not supported" error we are seeing could be related to fuse not setting the MS_POSIXACL flag?

Perhaps the only way to really get to the bottom of this problem would be for Netgear to provide you access to a debug environment (preferably built with full debug information and source code) so that you can see exactly what is happening. Would you be willing to try this if they could make such an environment available?

Thanks,

Mark


Tue Oct 19, 2010 18:07
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
Another idea - is there a way to turn on debug logging for FUSE and any of the other components that might be relevant at run-time? Perhaps that would help to identify the cause of the problem.


Tue Oct 19, 2010 18:14
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi again,

[replying to several posts, and trying to catch up with the latest one]

Quote:
There are a couple of things that seem strange to me.
1) Why is it that /dev/sda1 does not have an entry in /etc/fstab? Does NTFS-3G not use fstab?

ntfs-3g receives fstab options when present. Some kernel utility reads fstab to issue mounts, which may activate ntfs-3g. But the fstab entries have to be inserted (usually, either manually or by the kernel installer). Pluggable devices are generally mounted by hal or udev when the device is plugged.
Quote:
2) The entry for /dev/sda1 from mount does not contain the options acl,user_xattr even though I explicitly specified both options in the mount command as follows:

This is normal. These options are unknown to fuse, ntfs-3g processes them by itself. More precisely acl is not used, protections are governed by the presence of a UserMapping file (see http://www.tuxera.com/community/ntfs-3g ... s/#options )
Quote:
I noticed that the filesystem mounted at / (/dev/hdc1) also does not specify acl,user_xattr. So I tried to copy a file containing ACLs to /tmp and it fails too

Well, this is on ext3 and the acl option is meaningful. With no acl option on ext3 you are not supposed to be able to set ACLs (same for user_xattr, are you using extended attributes ?)
Quote:
When I copy the same file to the NTFS filesystem I get the identical error (which makes me wonder whether the lack of acl,user_xattr from mount in both cases may be a factor here) :

On ntfs-3g the acl option is not used (user_xattr is, even if not shown in mount, but this is not related to permissions). For ACLs, I repeat : you need a user mapping file, which you did not have, as the debug log states :
Quote:
Ownership and permissions disabled, configuration type 7

Quote:
I also repeated the same test adding the option "permissions". The result was identical to the result where I excluded "permissions".

The option permissions enables permissions and disables ACLs. This is not what you need.
Quote:
Note that I am getting "Operation not supported" even if I mount using the option "permissions". I think you told me before that if "permissions" is specified in the mount command then .NTFS-3G/UserMapping is not required and NTFS-3G will behave exactly as if "::S-1-5-21-3141592653-589793238-462643383-10000" were placed in this file. Is this correct?

This is true if you only want permissions, without the ACLs. This is clear in the log :
Quote:
User mapping built, Posix ACLs not used, configuration type 7

Quote:
Another interesting thing is that when using the "permissions" option, getfacl now returns something reasonable (it didn't work when we were using the UserMapping file

Well, it did. At least this was my conclusion after reading your post dated Sep 15, 19:43 Just note this does not prove the ACLs were processed, because there is a fall back on plain permissions.
Quote:
So it seems to me that the "permissions" option works differently (and seems to work slightly better) than the UserMapping approach.

It should be the same when using permissions and not using ACLs. The permission option is limited to hard-coded SIDs, which is probably not best for your needs.

Quote:
The symptoms of the poster's problem sound very similar to my problem. I wonder whether the "Operation not supported" error we are seeing could be related to fuse not setting the MS_POSIXACL flag?

I do not think this is related. I have not seen this issue myself, must be something specific to NFS.
Quote:
Perhaps the only way to really get to the bottom of this problem would be for Netgear to provide you access to a debug environment (preferably built with full debug information and source code) so that you can see exactly what is happening. Would you be willing to try this if they could make such an environment available?

Why not, if Netgear agrees. Hopefully this will not be needed.
Quote:
I wonder whether the "Operation not supported" error we are seeing could be related to fuse not setting the MS_POSIXACL flag?

Most of the "Operation not supported" you have shown are related to not having defined the user mapping. Be sure to have "User mapping built, Posix ACLs in use" in your debug log. Even now, fuse does not support the Posix ACLs, they are entirely supported by ntfs-3g.
Quote:
Another idea - is there a way to turn on debug logging for FUSE and any of the other components that might be relevant at run-time? Perhaps that would help to identify the cause of the problem.

None that I can think of. I think this is caused by the endianness of the interface, and such an error is considered an application error, not a ntfs-3g error, so it is not logged, and you have seen that the error code returned is ignored. You may get some confirmation by issuing a correct setfacl, and checking in the log whether the SETXATTR returns with an error EINVAL (-22)
Code:
# setfacl example, be sure there is a user mapping file
setfacl -m 'u::7,g::7,o::5,u:1234:7' <some ntfs file>


Regards

Jean-Pierre


Tue Oct 19, 2010 19:23
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
I have asked Netgear to try to perform a test using the latest version of NTFS-3G. Please could you check that I have provided the correct instructions?
http://www.readynas.com/forum/viewtopic ... 59#p265559

jpa wrote:
Note that the sync option was not implemented in ntfs-3g-2010.8.8, it has just been implemented and it is available in ntfs-3g-2010.10.2AR.1

Is this why I am also seeing the following error in the debug log file:
Code:
FLUSH[0]
   unique: 18, error: -90 (Function not implemented), outsize: 16


I think I must be a bit confused about the purpose of the "permissions" mount option. You see, I don't think I []ireally[/i] need ACLs to be copied by the cp -p command. In fact it would be useful if there were some way to tell cp to copy ownership and permissions but NOT to copy ACLs. It's only because of some bizarre behaviour from Microsoft Word (when a user saves a file that was previously created by a different user) that ACLs are even written to ext3 in the first place. Since my users (karen, mshields) both belong to the group "users" and the permission is 660 Samba should allow either karen or mshields to modify the file regardless of the ACLs (which is fine by me). That's why I have been using the "permissions" option during most of my testing. I realize that this might give me a problem when connecting the NTFS drive to a PC, but I thought that perhaps we should try to solve the "Operation Not Supported" problem before worrying about anything else.

Just to recap, my goals are as follows:
A) I don't want an "Operation not supported" (or any other) error message from cp -p every time the ext3 source file happens to contain an ACL
B) I do want the destination file (copy) on NTFS to have the correct Linux standard permissions and ownership (so that I can restore from NTFS to EXT3 if required)
C) I want the contents of the NTFS drive to be readable on any PC in the home

My understanding of the UserMapping file is that if it contains "::S-1-5-21-3141592653-589793238-462643383-10000" then it should achieve A and B if everything is working correctly. I also thought that the "Permissions" option should also be able to achieve A and B without any UserMapping file - why is this not the case? Are you saying that it is correct behaviour to get back "Operation not supported" when running in "permissions" mode when the file being copied happens to contain ACLs, and if so then why is that considered correct behaviour (I thought "permissions" mode just means "include ownership and standard permissions but filter out ACLs")?

A while back I asked if there was a way to achieve my goals without a UserMapping file and you responded:
jpa wrote:
Quote:
2) A way to avoid the need for the UserMapping file. Ideally users could just plug in a USB NTFS external drive and start using it immediately without having to worry about creating any UserMapping file. Is there any reason why NTFS-3G couldn't be changed so that by default, in the absence of any UserMapping file, it behaves as if the file contains "::S-1-5-21-3141592653-589793238-462643383-10000"?

That has be done for users not using ACLs (just use the option "permissions"), but so far I do not know of a single user having used it. I will do the same for ACL users if there is a need.

I am a user who don't intend to use or even need ACLs but they sometimes end up in some of my files (not by my choice). So that's why I thought I should use "Permissions" and not "UserMapping".

You also mentioned:
jpa wrote:
You can prevent ntfs-3g from using the ACL by mounting with option "permissions" (even if you have compiled with --enable-posix-acls). But this will probably not change anything, "cp -p" will still complain for not being able to copy the ACL.

Indeed "cp -p" does still complain, but why does "cp -p" even know that the ACL wasn't copied? If NTFS-3G has been configured in "permissions" mode, shouldn't NTFS-3G just report "success" (rather than throwing an EPERM error) back to "cp -p" even if ACLs were discarded?

It seems that I misunderstood what you were saying before and the correct interpretation (please confirm) is that successful copying of files without errors under all circumstances cannot be achieved with the "permissions" option ("permissions" mode will always generate an error if the file happens to contain an ACL for some reason). The only way that can work with all files (ACL or no ACL) is the "UserMapping" approach since you say
jpa wrote:
The reason why you get "Operation not supported" is you have not defined in .NTFS-3G/UserMapping the SIDs to be used, so no protection can be enabled.


So I repeated the "cp -p" test, this time with a "UserMapping" file and without the "permissions" option. Now I get "error: -22 (Invalid argument)" as well as the former "Function not implemented" error.
Code:
utime_omit_ok: 0
Version 2010.8.8 integrated FUSE 28
Mounted /dev/sda1 (Read-Write, label "OneTouch4", NTFS 3.1)
Cmdline options: debug,acl,user_xattr,rw,sync,noatime,allow_other,blksize=4096
Mount options: acl,rw,sync,allow_other,allow_other,nonempty,noatime,fsname=/dev/sda1,blkdev,blksize=4096
User mapping built, Posix ACLs in use, configuration type 7
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.8
flags=0x00000003
max_readahead=0x00020000
   INIT: 7.8
   flags=0x00000041
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, error: 0 (Success), outsize: 40
unique: 2, opcode: LOOKUP (1), nodeid: 1, insize: 49
LOOKUP /marktest
   NODEID: 2
   unique: 2, error: 0 (Success), outsize: 136
unique: 3, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 3, error: 0 (Success), outsize: 112
unique: 4, opcode: GETATTR (3), nodeid: 2, insize: 40
   unique: 4, error: 0 (Success), outsize: 112
unique: 5, opcode: LOOKUP (1), nodeid: 2, insize: 52
LOOKUP /marktest/INVOICE.doc
   NODEID: 3
   unique: 5, error: 0 (Success), outsize: 136
unique: 6, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 6, error: 0 (Success), outsize: 112
unique: 7, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 7, error: 0 (Success), outsize: 112
unique: 8, opcode: OPEN (14), nodeid: 3, insize: 48
   unique: 8, error: 0 (Success), outsize: 32
OPEN[0] flags: 0x40001 /marktest/INVOICE.doc
unique: 9, opcode: GETATTR (3), nodeid: 3, insize: 40
   unique: 9, error: 0 (Success), outsize: 112
unique: 10, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 0
   WRITE[0] 16384 bytes
   unique: 10, error: 0 (Success), outsize: 24
unique: 11, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 16384
   WRITE[0] 16384 bytes
   unique: 11, error: 0 (Success), outsize: 24
unique: 12, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 32768
   WRITE[0] 16384 bytes
   unique: 12, error: 0 (Success), outsize: 24
unique: 13, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 49152
   WRITE[0] 16384 bytes
   unique: 13, error: 0 (Success), outsize: 24
unique: 14, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 65536
   WRITE[0] 16384 bytes
   unique: 14, error: 0 (Success), outsize: 24
unique: 15, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 81920
   WRITE[0] 16384 bytes
   unique: 15, error: 0 (Success), outsize: 24
unique: 16, opcode: WRITE (16), nodeid: 3, insize: 16448
WRITE[0] 16384 bytes to 98304
   WRITE[0] 16384 bytes
   unique: 16, error: 0 (Success), outsize: 24
unique: 17, opcode: WRITE (16), nodeid: 3, insize: 11840
WRITE[0] 11776 bytes to 114688
   WRITE[0] 11776 bytes
   unique: 17, error: 0 (Success), outsize: 24
unique: 18, opcode: FLUSH (25), nodeid: 3, insize: 64
FLUSH[0]
   unique: 18, error: -90 (Function not implemented), outsize: 16
unique: 19, opcode: RELEASE (18), nodeid: 3, insize: 64
RELEASE[0] flags: 0x40001
   unique: 19, error: 0 (Success), outsize: 16
unique: 20, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 20, error: 0 (Success), outsize: 112
unique: 21, opcode: SETXATTR (21), nodeid: 3, insize: 124
   unique: 21, error: -22 (Invalid argument), outsize: 16
unique: 22, opcode: SETATTR (4), nodeid: 3, insize: 128
   unique: 22, error: 0 (Success), outsize: 112


jpa wrote:
On Sep 15th you have reported that you could copy ownership and permissions after having defined a standard user mapping (though the ACL issue is not yet solved in your configuration)

Yes, I reported that with UserMapping mode ownership and standard permissions are ok, BUT getfacl doesn't print any useful information at all (see immediately below). So in my view the "UserMapping" option seems to be working even worse for me than the "permissions" option
Code:
# getfacl /tmpusb/marktest/INVOICE.doc
getfacl: /tmpusb/marktest/INVOICE.doc: Success

jpa wrote:
Apart from the ACLs, you can already backup and restore on Linux

But I get "Operation not supported" errors if I use "Permissions" mode and I get "Invalid Argument" errors if I use "UserMapping" mode. So there is no mode where I am able to run a successful error-free backup to NTFS.
jpa wrote:
The main problem is that your files show no permissions for "other" (neither owner nor group). This is what prevents from copying on any Windows computer. Is this intentional in your security policy ?

I chose to make files unreadable by "other" on Linux in case I ever add other users to the NAS who should not have access to the files on the lifepractice share. I suppose I could relax the permission for "other" so that files are readable if there is no alternative. But before we get to that I think it would be good to solve these "Operation not supported" and "Invalid Argument" error messages.

By the way, here is some additional info about the version of fuse on the machine in case it's helpful.
Code:
tera:/lib/modules/2.6.17.8ReadyNAS/kernel/fs/fuse# ls -lt --full-time fuse.ko   
-rw-r--r--    1 root     root       712723 Tue Jun 09 21:59:51 2009 fuse.ko

srcversion=62E4EED50E149E5676352B4


Tue Oct 19, 2010 22:41
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
Hi Jean-Pierre,

I'm also going to try the command you suggested:
jpa wrote:
You may get some confirmation by issuing a correct setfacl, and checking in the log whether the SETXATTR returns with an error EINVAL (-22)
Code:
# setfacl example, be sure there is a user mapping file
setfacl -m 'u::7,g::7,o::5,u:1234:7' <some ntfs file>



The result (using a UserMapping file) is
Code:
# setfacl -m 'u::7,g::7,o::5,u:1234:7' /tmpusb/marktest/test
setfacl: /tmpusb/marktest/test: Success

and the debug log contains
Code:
unique: 55, opcode: LOOKUP (1), nodeid: 1, insize: 49
LOOKUP /marktest
   NODEID: 2
   unique: 55, error: 0 (Success), outsize: 136
unique: 56, opcode: LOOKUP (1), nodeid: 2, insize: 45
LOOKUP /marktest/test
   NODEID: 4
   unique: 56, error: 0 (Success), outsize: 136
unique: 57, opcode: GETATTR (3), nodeid: 4, insize: 40
   unique: 57, error: 0 (Success), outsize: 112
unique: 58, opcode: LOOKUP (1), nodeid: 1, insize: 49
LOOKUP /marktest
   NODEID: 2
   unique: 58, error: 0 (Success), outsize: 136
unique: 59, opcode: GETXATTR (22), nodeid: 4, insize: 72
   unique: 59, error: 0 (Success), outsize: 44


Regards,

Mark


Tue Oct 19, 2010 22:47
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
Also, when I remove the .NTFS-3G directory, unmount, then remount, getfacl starts working again
Code:
# getfacl /USB_HDD_5/marktest/test
getfacl: Removing leading '/' from absolute path names
# file: USB_HDD_5/marktest/test
# owner: root
# group: root
user::rwx
group::rwx
other::rwx

tera:/c/lifepractice/marktest# ls -l  /USB_HDD_5/marktest/test
-rwxrwxrwx    1 root     root            0 Oct 19 15:11 /USB_HDD_5/marktest/test


But I can't see any evidence that the prior setfacl command worked (the file permission is 777 not 755).


Tue Oct 19, 2010 23:05
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi Mark,
Quote:
I have asked Netgear to try to perform a test using the latest version of NTFS-3G. Please could you check that I have provided the correct instructions?

As far as I can see, these instructions are correct. Two remarks though :
Code:
setfacl -m u:mark:rw- test

This is assuming there is a "mark" user in his environment, better use a numeric uid (eg 1234)
Code:
ntfs-3g -o debug,acl,user_xattr,rw,sync, [...]

Where did you get instructions to use the acl option ? This is meaningless for ntfs-3g (but it should not hurt).
Quote:
I think I must be a bit confused about the purpose of the "permissions" mount option. You see, I don't think I []ireally[/i] need ACLs to be copied by the cp -p command. In fact it would be useful if there were some way to tell cp to copy ownership and permissions but NOT to copy ACLs.

The manual for cp is not clear about it, but there is no apparent option for copying the basic permissions and not copy the ACLs.
rsync is smarter, with "rsync -opg" permissions and ownership are copied, and there is another option -A for copying the ACLs.
Did you try hijacking rsync (through alias or PATH) in order to force appropriate options ?
Quote:
It's only because of some bizarre behaviour from Microsoft Word (when a user saves a file that was previously created by a different user) that ACLs are even written to ext3 in the first place.

More likely this is caused by Samba which has to deal with differences of representation of permissions across machines.
Quote:
but I thought that perhaps we should try to solve the "Operation Not Supported" problem before worrying about anything else.

More on this below.
Quote:
A) I don't want an "Operation not supported" (or any other) error message from cp -p every time the ext3 source file happens to contain an ACL

This requires fixing ntfs-3g, I hope this is fixed in the proposed version.
Quote:
B) I do want the destination file (copy) on NTFS to have the correct Linux standard permissions and ownership (so that I can restore from NTFS to EXT3 if required)
C) I want the contents of the NTFS drive to be readable on any PC in the home

B) and C) are contradictory if you want your files on Linux to be unreadable by anybody other than owner and group.
Quote:
My understanding of the UserMapping file is that if it contains "::S-1-5-21-3141592653-589793238-462643383-10000" then it should achieve A and B if everything is working correctly.

Yes, I think so.
Quote:
I also thought that the "Permissions" option should also be able to achieve A and B without any UserMapping file - why is this not the case? Are you saying that it is correct behaviour to get back "Operation not supported" when running in "permissions" mode when the file being copied happens to contain ACLs,

Yes. with permissions you are supposed to get the same behaviour as on ext3 (and not setting the acl option when mounting ext3).
Quote:
why is that considered correct behaviour (I thought "permissions" mode just means "include ownership and standard permissions but filter out ACLs")?

Because it is that way on other file systems. "permissions" means "include ownership and permission and reject ACLs".
Quote:
I am a user who don't intend to use or even need ACLs but they sometimes end up in some of my files (not by my choice). So that's why I thought I should use "Permissions" and not "UserMapping".

"permissions" is a simple way to get ownership and permissions with limited access from Windows (because the created SIDs are hardcoded and different from what a specific Windows PC expect). With a UserMapping file (which takes precedence over most permission options) you can indicate the SID you want for each Linux user and group, thus you can have a perfect interpretation of ownership, permissions and ACLs on a Windows PC which knows about the mentioned SIDs. The problem here is you have several Windows PC with the same users and different SIDs.
Quote:
Indeed "cp -p" does still complain, but why does "cp -p" even know that the ACL wasn't copied? If NTFS-3G has been configured in "permissions" mode, shouldn't NTFS-3G just report "success" (rather than throwing an EPERM error) back to "cp -p" even if ACLs were discarded?

"cp -p" complains because it tries to copy the ACLs and cannot do so. In permissions mode, it gets a (rightful) "operation not supported", in ACL mode it gets an "Invalid argument" error (ntfs-3g bug, probably caused by unexpected endianness)
Quote:
It seems that I misunderstood what you were saying before and the correct interpretation (please confirm) is that successful copying of files without errors under all circumstances cannot be achieved with the "permissions" option ("permissions" mode will always generate an error if the file happens to contain an ACL for some reason).

Correct. "permissions" imply rejecting ACL operations (getfacl falls back on basic permissions)
Quote:
So I repeated the "cp -p" test, this time with a "UserMapping" file and without the "permissions" option. Now I get "error: -22 (Invalid argument)" as well as the former "Function not implemented" error.

The "Invalid argument" is a bug, and the "Function not implemented" is caused by the use of the "sync" option, which is not implemented in the version you are using. This is not related to permissions issues.
By the way, the sync option will cause more IOs and reduce performances.
Quote:
But I get "Operation not supported" errors if I use "Permissions" mode and I get "Invalid Argument" errors if I use "UserMapping" mode. So there is no mode where I am able to run a successful error-free backup to NTFS.

I have a proposal : there is an option "silent" which currently means "do not return not supported for chmod and chown with no user mapping available". I can probably extend this option to cover getfacl and setfacl, but before that I have to make tests to know how this will interact with falling back on protections.
This would hide the "not implemented" error with "cp -p" and no ACLs, but will not help in reading your backup files directly from Windows when these files are marked as only readable by owner and group.
Quote:
By the way, here is some additional info about the version of fuse on the machine in case it's helpful.

You fuse.ko was compiled in June 2009, but you have an older kernel module. In the debug log you see "INIT: 7.8", this is a negotiated fall back to an older version. Current versions with ACLs enabled would show at least 7.12.
Quote:
Also, when I remove the .NTFS-3G directory, unmount, then remount, getfacl starts working again

Well, not really, you are getting the result of the fall back on permissions, which fall back on no protections as permissions are not enabled.

Hope all of this is useful, you have a lot of constraints, and you will probably have to drop some of them.

Regards

Jean-Pierre


Wed Oct 20, 2010 10:17
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi again,

Here are my test results for a suggested "silent" option which would cause the "non implemented" errors to be hidden when setting an ACL in circumstances where this cannot be implemented (rejected by option, no user mapping, etc.)

Behavior of "cp -p"
1) chown
2) set the ACL
3) if no error, stop there, else do a chmod

Behavior of "rsync -opgA"
1) chmod to a temporary 700
2) get the current ACL of the file
3) chown
4) set the ACL (if different from the one got at step 2)
5) if no error, stop there, else do a chmod

In both cases returning with no error from ACL setting implies not copying the permissions.

This is not what you ask for, so I have to drop the proposal.

Regards

Jean-Pierre


Wed Oct 20, 2010 11:13
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
jpa wrote:
Code:
ntfs-3g -o debug,acl,user_xattr,rw,sync, [...]

Where did you get instructions to use the acl option ? This is meaningless for ntfs-3g (but it should not hurt).

I think I found it through googling as a potential solution for fixing "Operation not supported" issues on ext3 but as you say NTFS-3G does not use this option but instead uses the presence/lack-of a UserMapping file to switch ACL mode on/off. Please can you tell me how "user_xattr" mode is determined in NTFS-3G and what effect it has (if any)?
jpa wrote:
Because it is that way on other file systems. "permissions" means "include ownership and permission and reject ACLs".

I see. Thanks for clarifying this point.

Please can I clarify a few more things about the "permissions" option. According to the manpage (http://linux.die.net/man/2/setxattr) SETXATTR is for setting extended attributes. I can see that I don't have any extended attributes defined for the file "test" that I was copying.
Code:
# getfattr -d test
(Produces no output)

Despite the lack of extended attributes, I notice that even in the case of "permissions" mode, SETXATTR is still being called. Why is SETXATTR being called?
Quote:
unique: 22, opcode: SETXATTR (21), nodeid: 3, insize: 124
unique: 22, error: -45 (Operation not supported), outsize: 16


jpa wrote:
Quote:
B) I do want the destination file (copy) on NTFS to have the correct Linux standard permissions and ownership (so that I can restore from NTFS to EXT3 if required)
C) I want the contents of the NTFS drive to be readable on any PC in the home

B) and C) are contradictory if you want your files on Linux to be unreadable by anybody other than owner and group.

I guess I don't know enough about how you are embedding the Linux ownership & permissions data within the files and directories on the NTFS file system. Presumably you could store pretty much any data you want in an Alternate Data Stream (http://linux.die.net/man/8/mount.ntfs-3g). If you stored all the Linux permissions and ownership in an ADS that Windows never looks at then couldn't you completely decouple the Windows ACLs from the Linux ownership & permissions?

The other thing that I wondered about was whether it might be possible to add an ACL for a special Windows "well-known security identifier" to allow full acceess:
http://support.microsoft.com/kb/243330

What would happen if you granted full 777 access to SID: S-1-1-0 (Everyone) - would this allow Windows full access without affecting Linux?

jpa wrote:
mjw wrote:
Also, when I remove the .NTFS-3G directory, unmount, then remount, getfacl starts working again

Well, not really, you are getting the result of the fall back on permissions, which fall back on no protections as permissions are not enabled.

I see, but this still doesn't explain why in "UserMapping" mode getfacl is unable to print out an ACLs, yet still seems to think it succeeded. Do you expect this will also be resolved by big-endian fixes in the latest version of NTFS-3G?
jpa wrote:
In both cases returning with no error from ACL setting implies not copying the permissions.

This is not what you ask for, so I have to drop the proposal.

So are you saying that this behaviour (of first trying to set the ACL then, only if the ACL setting failed, then finally fallback and try to chmod) is hardcoded within cp and rsync so there is no way to solve the problem though the "silent" option? If this were the case, then surely this would cause "cp -p" to always generate an error message on a non-ACL filesystem even when copying files that contain no ACLs at all. I can see that this does not happen in /tmp (which does not support ACLs):
Code:
cd /tmp
tera:/tmp# touch test
tera:/tmp# ls -l test
-rw-r--r--    1 root     root            0 Oct 20 19:53 test
tera:/tmp# chown mark test
tera:/tmp# cp -p test test2
(No error message is printed)

This indicates to me that on a non-ACL filesystem (e.g. "permissions" mode) as long as NTFS-3G don't reveal the existence of ACLs in a file to the cp command (e.g. getfacl just returns the fallback ACLs regardless of what the real ACLs are), cp won't output any error message. Does this sound plausible?

jpa wrote:
Hope all of this is useful, you have a lot of constraints, and you will probably have to drop some of them.

Yes, thanks very much for all your help it is most useful and much appreciated. I really hope this this ends up being of use to you as well. By the way, I found a presentation that seems to cover a lot of the issues around SID mapping:
http://www.linuxplusvalue.be/download/CIFSnPOSIX.pdf
It seems there are various approaches to getting the SID. In addition to Winbind there are also LDAP and ActiveDirectory solutions but it seems that the latter solutions are more suited when you have a larger enterprise environment.

I have some hope that we may solve the ACL problems in due course, but as you say, the problem of how to make the files readable on Windows will still remain. There are a few potential workarounds for this problem:
1. Make files readable by "other"
2. Find some sneeky trick to make the files accessible to Windows without affecting Linux permissions & ownership (perhaps using a "well-known security identifier" or by somehow hiding the Linux permissions from the Windows permissions as suggested above)
3. Format the external drive as ext3 then use Ext2IFS (http://www.fs-driver.org/) or Ext2FSD (http://www.ext2fsd.com/?page_id=2) to allow Windows to read the drive. The advantages of this approach are that it perfectly retains Linux permissions and ownership, gives the highest performance copying, while also allowing Windows to read any file. The disadvantage is that it requires additional software to be installed on each Windows PC. Also, neither package seems to work on Windows-7 64-bit as far as I can tell.
4. Attach 2 external backup drives (one ext3 and one NTFS). Use ext3 to preserve the Linux ownership & permissions plus NAS recovery and use NTFS to allow Windows users to be able to quickly access files in an emergency

In the past I (and the majority of Readynas users) have been using option 3, but I would be very disappointed if we don't find a better solution using NTFS-3G after all the effort we have both put in. Option 1 sounds like a possible compromise (although not ideal).

jpa wrote:
rsync is smarter, with "rsync -opg" permissions and ownership are copied, and there is another option -A for copying the ACLs.
Did you try hijacking rsync (through alias or PATH) in order to force appropriate options ?

Thanks - this was a brilliant idea. The current Readynas backup tool uses the command "rsync -aAv" which of course results in lost of "Operation not supported" errors. I wrote a quick script, and put it in /usr/local/bin/rsync (the real rsync is in /usr/bin/rsync)
Code:
#!/bin/bash
#The purpose of this script is to intercept rsync commands and replace the -aAv
#so that ownership and permissions are copied but ACLs are not
#
#echo "rsync called with arguments " $@
MODIFIED_OPTIONS=`echo $@ | sed  's/-aAv/-avopg/g'`
/usr/bin/rsync $MODIFIED_OPTIONS


I have confirmed that this does exactly what I need:
- Retains permissions and ownership
- Doesn't output any errors (so this solves my immediate deluge of nightly emails for backup failures that aren't serious failures)

It also works in both "permissions" mode and "UserMapping" mode. So now at least I have a workaround (all-be-it kludgey) so I don't get emailed about backup failures that weren't actual backup failures that I would need to worry about.

However, although this fix is probably sufficient for me for the interim, this fix is not a general/complete fix for the following reasons:
- It not only prevents the copying of ACLs for NTFS file systems, it prevents copying for all file systems (including ext3)
- It doesn't handle the alternate backup command supported by the Readynas that uses "cp -p"
- It doesn't solve the problem of being able to access the files from any Windows PC

It will be good to see what feedback we get from Netgear regarding your new version of NTFS-3G.


Wed Oct 20, 2010 21:20
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
I also thought of another idea regarding how to make the external NTFS drive readable by anyone without compromising the security of the main NAS ext3 file system. Perhaps if there was a new type of inverse_file_mask that rather than being AND'ed with the permissions was instead OR'ed with the permissions we could ensure that files copied from ext3 to NTFS always have read permission for "other".

For example
Code:
inverse_file_mask=001     # Ensures that file always has read permission by other

We would probably need a separate inverse_directory_mask as well:
Code:
inverse_directory_mask=005     # Ensures that directory is always accessible by other


It sounds like Samba does a similar thing with "force security mode"
http://www.samba.org/samba/docs/man/Sam ... #id2614099


Wed Oct 20, 2010 21:41
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi Mark,

Quote:
Please can you tell me how "user_xattr" mode is determined in NTFS-3G and what effect it has (if any)?

It works like on ext3, however this is by default on ntfs-3g (which you can override by using the streams_interface option). So my question is do you use user-defined extended attributes ?
Quote:
According to the manpage (http://linux.die.net/man/2/setxattr) SETXATTR is for setting extended attributes. I can see that I don't have any extended attributes defined for the file "test" that I was copying.

The extended attributes and the Posix ACLs were developped by the same team, and the same interfaces are used by both. However the ACLs use a name space different from user-defined extended attributes, and they are processed quite differently in ntfs-3g. The user_xattr option has no effect on ACL support (see http://www.tuxera.com/community/ntfs-3g ... ttributes/ )
Quote:
I guess I don't know enough about how you are embedding the Linux ownership & permissions data within the files and directories on the NTFS file system. Presumably you could store pretty much any data you want in an Alternate Data Stream (http://linux.die.net/man/8/mount.ntfs-3g). If you stored all the Linux permissions and ownership in an ADS that Windows never looks at then couldn't you completely decouple the Windows ACLs from the Linux ownership & permissions?

The main design option in ntfs-3g is to use NTFS the same way as Windows, in order to get the best interoperability, and to be sure Microsoft cannot change the specifications without ruling out its own products. In particular when you set permissions on Linux, you write the same data as Windows would have done for the permissions you requested. However the underlying concepts are different, so this has to be the best approximation.
The ADS are reserved for extended attributes. When you create an extended attribute through Java in Windows, and later do the same thing on Linux and ntfs-3g you write the same data at the same place.
Quote:
The other thing that I wondered about was whether it might be possible to add an ACL for a special Windows "well-known security identifier" to allow full acceess:

I have made interesting tries. I am probably able to define an option "windows-readable" which would force an extra ACL for Windows to be able to read any file (in short this is a Trojan horse).
Quote:
What would happen if you granted full 777 access to SID: S-1-1-0 (Everyone) - would this allow Windows full access without affecting Linux?

But, of course granting full access to SID: S-1-1-0 is the same as "chmod o+rwx" which ntfs-3g translates to Windows concepts (this is interoperability).
Quote:
I see, but this still doesn't explain why in "UserMapping" mode getfacl is unable to print out an ACLs, yet still seems to think it succeeded. Do you expect this will also be resolved by big-endian fixes in the latest version of NTFS-3G?

You are using an early and incomplete implementation of ACLs in Linux. You have already noticed that "ls" does not show the presence of ACLs on ext3. The ACL library is probably disturbed by the "Invalid argument" error. I expect this to be solved by the endianness fix, until some actual check proves wrong. I have no Linux big-endian computer.
Quote:
So are you saying that this behaviour (of first trying to set the ACL then, only if the ACL setting failed, then finally fallback and try to chmod) is hardcoded within cp and rsync so there is no way to solve the problem though the "silent" option?

Yes.
Quote:
If this were the case, then surely this would cause "cp -p" to always generate an error message on a non-ACL filesystem even when copying files that contain no ACLs at all. I can see that this does not happen in /tmp (which does not support ACLs):

When there is no ACL to set, cp does a plain chmod. This is why you got no error when backuping the files with no ACL.
Quote:
This indicates to me that on a non-ACL filesystem (e.g. "permissions" mode) as long as NTFS-3G don't reveal the existence of ACLs in a file to the cp command (e.g. getfacl just returns the fallback ACLs regardless of what the real ACLs are), cp won't output any error message. Does this sound plausible?

Yes, when there is no ACL, cp does not use setfacl so it does not get any error. My remarks were related to files with an ACL and setfacl returning an error, triggering a need for a fall back.
Quote:
It seems there are various approaches to getting the SID. In addition to Winbind there are also LDAP and ActiveDirectory solutions but it seems that the latter solutions are more suited when you have a larger enterprise environment.

I am much interested, but I know of no real user wanting to cooperate.
Quote:
1. Make files readable by "other"

That would be clean, but you also want the files not to be readable by other than owner and group.
Quote:
2. Find some sneeky trick to make the files accessible to Windows without affecting Linux permissions & ownership (perhaps using a "well-known security identifier" or by somehow hiding the Linux permissions from the Windows permissions as suggested above)

I may have a possibility for such a trick. More test and development needed.
Quote:
3. Format the external drive as ext3 then use Ext2IFS

Maybe, I have not tried.
Quote:
while also allowing Windows to read any file.

So it does not respect the permissions... ntfs-3g strives to do better interoperability.
Quote:
4. Attach 2 external backup drives (one ext3 and one NTFS). Use ext3 to preserve the Linux ownership & permissions plus NAS recovery and use NTFS to allow Windows users to be able to quickly access files in an emergency

You would have a safer backup, but more complex management.
Quote:
However, although this fix is probably sufficient for me for the interim, this fix is not a general/complete fix for the following reasons:
- It not only prevents the copying of ACLs for NTFS file systems, it prevents copying for all file systems (including ext3)

You are surely able to improve your script to set the options according to the destination path.
Quote:
It will be good to see what feedback we get from Netgear regarding your new version of NTFS-3G.

Part of this discussion is void if the version cannot be compiled...
Quote:
Perhaps if there was a new type of inverse_file_mask that rather than being AND'ed with the permissions was instead OR'ed with the permissions we could ensure that files copied from ext3 to NTFS always have read permission for "other".

This is probably possible, but it obviously requires more development.

What would be your emergency restore procedure ? Does it have to be made by a non-Administrator user doing copy and paste from an USB device ? Can it be made through a cmd script run by a local Administrator ?

Also : I saw on the Netgear forum that "ewok" got an "operation not supported" error. As far as I can tell this is not a bug. This is some action not possible with the current configuration. Most likely he has not created a .NTFS-3G/UserMapping file (or created at a wrong location, or with wrong contents). Be sure the debug log shows "User mapping built, Posix ACLs in use, configuration type 7"

The buggy situation leads to an "invalid argument" error.

Regards

Jean-Pierre


Thu Oct 21, 2010 09:53
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
jpa wrote:
So my question is do you use user-defined extended attributes ?

How can I check this? I tried
Code:
getfattr -d <file>

Some of my music files contain
Code:
user.DOSATTRIB="0x20"

Is this a user extended attribute? If so, then yes I am using it, but I'm not sure whether I really need it. For some music files when viewing in Windows Explorer I can see additional metadata like duration, artist, genre, etc. Perhaps this also uses the extended attribute mechanism.
jpa wrote:
The main design option in ntfs-3g is to use NTFS the same way as Windows, in order to get the best interoperability ...

Yes, I see this makes sense. Also you need to handle various use-cases such as dual-booting Linux and Windows on the same machine so I can understand why such a user would need the Linux and Windows permissions & ownership to be as closely synchronized as possible.
jpa wrote:
I have made interesting tries. I am probably able to define an option "windows-readable" which would force an extra ACL for Windows to be able to read any file (in short this is a Trojan horse).

That sounds promising, I would be very interested to hear if you find a workable solution (alternatively see my other attempts below using rsync --chmod).
jpa wrote:
I am much interested, but I know of no real user wanting to cooperate.

There's a good chance if you posted a request for assistance from one of the ReadyNAS users on the Netgear forum, someone running in "Domain" mode would be willing to run this command for you and send you the output (especially if you explain that you are trying to improve the NTFS security functionality)
Code:
net idmap dump /var/lib/samba/winbindd_idmap.tdb > idmap_dump.txt

jpa wrote:
mjw wrote:
1. Make files readable by "other"

That would be clean, but you also want the files not to be readable by other than owner and group.

Well, I'm thinking that I could configure the NAS so that the NTFS drive is not shared (visible) to any users (i.e. users can only access EXT3 via Samba). This way it wouldn't matter if the permissions on the external drive were less stringent than on ext3 since the only way that security could be compromised would be if someone unauthorized gained physical access to the external drive. But in that situation, no permissions/ownership/ACLs are going to help anyway since they could just copy the drive with a program that doesn't honour NTFS file permissions.

So if there is a way to make files NOT readable by "other" on ext3, but make the NTFS backup files readable by "other" then that would be a solution. Switching off all permissions & ownership options on NTFS-3G is a way to achieve this (but you lose permissions&ownership which is undesirable for restoring to ext3). When I do this, all files on NTFS are owned by "root" with full control. I suggested the "inverse_file_mask" as a potential solution but that would require development. I think there is a better solution by using the "--chmod" option in rsync
Code:
rsync -avopg --chmod=o+r

If you can think of another approach (especially one that would also work with "cp") then please let me know.
jpa wrote:
You are surely able to improve your script to set the options according to the destination path.

This is true. But I'm thinking that it should really be Netgear that fixes this for all their users. I'm thinking of asking Netgear to add an enhancement to their "Frontview" user interface to allow users to override the options they are passing to cp and rsync. This way it should be possible to avoid backup errors regardless of the mode of NTFS-3G (default, permissions, UserMapping).

Also (assuming that we solve the "invalid argument" error) I'm thinking of requesting that they provide some options to allow users to select whether to enable/disable the following on a per-filesystem basis (currently they support ext3, fat and ntfs):
- Ownership & Permissions
- ACL
- Extended Attributes
jpa wrote:
Part of this discussion is void if the version cannot be compiled...

I'm working on that - I've also asked the original developer who helped me. He has replied that he will compile for me once he has his Solaris box running again (hopefully soon). I also asked him if he would produce a idmap_dump.txt file for you but unfortunately he doesn't have any Windows machines in his environment.

jpa wrote:
What would be your emergency restore procedure ? Does it have to be made by a non-Administrator user doing copy and paste from an USB device ? Can it be made through a cmd script run by a local Administrator ?

I'm thinking that the simpler the better (since my sister is not so technical). The easiest would be if she just has to unplug the drive from the NAS and connect it to one of the 3 PCs in her house (2 Windows-XP and 1 Windows-7). If we can solve the "invalid argument" error then I might try to do things properly and setup a full UserMapping file to enable this. The idea of a cmd script might also be worth exploring especially if we can't get ACLs working.

jpa wrote:
Also : I saw on the Netgear forum that "ewok" got an "operation not supported" error. As far as I can tell this is not a bug. This is some action not possible with the current configuration. Most likely he has not created a .NTFS-3G/UserMapping file (or created at a wrong location, or with wrong contents). Be sure the debug log shows "User mapping built, Posix ACLs in use, configuration type 7"

The buggy situation leads to an "invalid argument" error.

Yes, I'm sure you are right. I've only just myself begun to understand the full implications of running in UserMapping mode so it's understandable if ewok missed that point slightly. I'm could definitely respond, but I'm thinking that it might be better if you were to respond to him on the Netgear forum. It might help communication if the developers involved can communicate directly without me getting in the way. What do you think?


Thu Oct 21, 2010 17:31
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi again,

Quote:
Some of my music files contain

user.DOSATTRIB="0x20"

Is this a user extended attribute? If so, then yes I am using it, but I'm not sure whether I really need it. For some music files when viewing in Windows Explorer I can see additional metadata like duration, artist, genre, etc. Perhaps this also uses the extended attribute mechanism.

Yes and no. DOSATTRIB is a Windows specific attribute which Windows associates to any file. But this kind of attribute does not exist in ext3, so Samba has translated it to a user-defined extended attribute named DOSATTRIB. If you were using "cp --preserve=xattr" the extended attribute would be copied as a user extended attribute to ntfs.... but this is not what Windows would have done. ntfs-3g does not unwind the transformations done by Samba. They can be unwound by a shell script, but you would have to do the opposite before restoring on ext3 (the value 0x20 for DOSATTRIB is a legacy flag meaning "this file has been changed since last backup").
If you have metadata associated to your multimedia files, they probably are true extended attributes, which can be copied by "cp --preserve=xattr" or "rsync -X".
Quote:
Quote:
I have made interesting tries. I am probably able to define an option "windows-readable" which would force an extra ACL for Windows to be able to read any file (in short this is a Trojan horse).

That sounds promising, I would be very interested to hear if you find a workable solution (alternatively see my other attempts below using rsync --chmod).

Still testing. The Trojan horse aspect is dangerous : on Linux you would set ACLs to protect a file, without being able to see that the protected file is readable by any plain user on any Windows computer (just imagine children using Windows to access files saved by their parents who use Linux on a dual-boot computer).
Quote:
There's a good chance if you posted a request for assistance from one of the ReadyNAS users on the Netgear forum, ...

Good idea, if they are cooperative in solving the current issue.
Quote:
Well, I'm thinking that I could configure the NAS so that the NTFS drive is not shared (visible) to any users (i.e. users can only access EXT3 via Samba). This way it wouldn't matter if the permissions on the external drive were less stringent than on ext3

I did not mean different protections on ext3 and ntfs, but you implicitly confirm you want access limited to owner and group on ext3 (and Samba protections do not help you achieving that).
Quote:
So if there is a way to make files NOT readable by "other" on ext3, but make the NTFS backup files readable by "other" then that would be a solution. Switching off all permissions & ownership options on NTFS-3G is a way to achieve this (but you lose permissions&ownership which is undesirable for restoring to ext3). When I do this, all files on NTFS are owned by "root" with full control. I suggested the "inverse_file_mask" as a potential solution but that would require development. I think there is a better solution by using the "--chmod" option in rsync

I have analyzed the consequences of options fmode=004 and dmode=005 to define protections to be "or-ed" in chmod and setxattr on files and directories. This is apparently doable and clean (several days needed). The consequence is you will have to do chmod's in your restore script on Linux, but the backup files will be readable on any Windows computer.
Quote:
This is true. But I'm thinking that it should really be Netgear that fixes this for all their users. I'm thinking of asking Netgear to add an enhancement to their "Frontview" user interface to allow users to override the options they are passing to cp and rsync. This way it should be possible to avoid backup errors regardless of the mode of NTFS-3G (default, permissions, UserMapping).

Allowing any option may be a problem for them (user complains about bad backups), but they can propose several safety options, as you suggest.
Quote:
I'm working on that - I've also asked the original developer who helped me. He has replied that he will compile for me once he has his Solaris box running again (hopefully soon).

There is no gcc cross-compiler for sparc on a standard PC ?
Quote:
I'm could definitely respond, but I'm thinking that it might be better if you were to respond to him on the Netgear forum. It might help communication if the developers involved can communicate directly without me getting in the way. What do you think?

The big thing is compiling for the sparc, and the Netgear guys know that best. Now the tests : see the number and length of posts we had to exchange. I hesitate to do the same with them, and they are probably not flooded by requests urging them to fix something. The first step is to get the new compiled version, then you retry, we analyze the results, and see what should be done next.

Regards

Jean-Pierre


Thu Oct 21, 2010 19:32
Profile

Joined: Mon Sep 13, 2010 18:42
Posts: 42
Post Re: Operation not supported (45) - ACL problem
Surprisingly it turns out that on Windows XP home, my sister was able to read a file copied to the NTFS drive (using "permissions" mode) with permission 0670. I don't know why, but it seems that it is not necessary to have permission for "other". If this is indeed true, then perhaps I have a full solution without any fixes to NTFS-3G simply by using "permissions" mode.


Thu Oct 21, 2010 19:32
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Operation not supported (45) - ACL problem
Hi again,

Quote:
Surprisingly it turns out that on Windows XP home, my sister was able to read a file copied to the NTFS drive (using "permissions" mode) with permission 0670. I don't know why, but it seems that it is not necessary to have permission for "other". If this is indeed true, then perhaps I have a full solution without any fixes to NTFS-3G simply by using "permissions" mode.

She is probably using an Administrator account (as do most home users). If not, please double-check.

Regards

Jean-Pierre


Thu Oct 21, 2010 19:47
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 79 posts ]  Go to page Previous  1, 2, 3, 4  Next


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.