FAQ SearchLogin
Tuxera Home
View unanswered posts | View active topics It is currently Thu Oct 28, 2021 10:38



Post new topic Reply to topic  [ 17 posts ] 
Feature request: Utility to fix geometry info & change UUID 
Author Message

Joined: Fri Dec 10, 2010 03:40
Posts: 6
Post Feature request: Utility to fix geometry info & change UUID
Hi,

at http://www.linux-ntfs.org/doku.php?id=contrib:ntfsreloc, there used to be a program called "ntfsreloc" which was able to change the "number of heads", "sectors per track" and "hidden sectors" values in the boot record. It was quite hard to find that it has been renamed and can now be found at http://sourceforge.net/projects/ntfsfixboot/.

Do you think it would be a good idea to integrate this functionality in the ntfsprogs package? mkntfs is able to do these settings at formatting time (and warns if they are not set), so I think it would be nice to have a utility in the package that can fix them later.

Another often-need feature is the ability to change the volume serial number. Several distributions (e.g. Ubuntu) use it as the UUID in /etc/fstab now, and in various support forums some rather dangerous scripts are floating around that don't seem to care about the backup boot sector.


Fri Dec 10, 2010 04:11
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi,

Quote:
Do you think it would be a good idea to integrate this functionality in the ntfsprogs package?

What for ? The number of heads and number of sectors per track are totally virtual nowadays. Aren't they fixed when resizing a partition ?

Quote:
Another often-need feature is the ability to change the volume serial number. Several distributions (e.g. Ubuntu) use it as the UUID in /etc/fstab now, and in various support forums some rather dangerous scripts are floating around that don't seem to care about the backup boot sector.

In an NTFS partition there are two different things : the volume serial number and the volume UUID. The former is 8 bytes long and stored in the boot sector (and backup boot sector), the latter is 16 bytes long and stored in the volume description.

Changing the UUID implies changing the OID of all files which have one (shortcuts, etc.) if you do not want to defeat the file tracking procedures. To me this is reasonable at formatting time only.

And using the volume UUID for "/dev/disk/by-uuid" instead of the serial number would enable tracking files on Linux.

Regards

Jean-Pierre


Fri Dec 10, 2010 09:54
Profile

Joined: Fri Dec 10, 2010 03:40
Posts: 6
Post Re: Feature request: Utility to fix geometry info & change UUID
Quote:
What for ? The number of heads and number of sectors per track are totally virtual nowadays. Aren't they fixed when resizing a partition ?

Windows XP needs the number of heads and sectors per track in order to boot. "man mkntfs" claims that (see the options "-H" and "-S"), and it's true for XP. I've experienced that myself on my ThinkPad, which has a BIOS reported geometry of 5168/240/63 (and not the 4864/255/63 that most partitioning tools guess). A partition table based on the "wrong" geometry works, but a wrong geometry in the NTFS boot sector prevents XP from booting.

These two things don't need to be fixed when a partition is resized, but sometimes when a partition is moved from one computer to another (e.g. from a Thinkpad to a "normal" computer or vice versa). That's why e.g. partclone includes a copy of this program.

In addition, the "hidden sectors" value needs to be fixed when a partition is moved.
Quote:
In an NTFS partition there are two different things : the volume serial number and the volume UUID. The former is 8 bytes long and stored in the boot sector (and backup boot sector), the latter is 16 bytes long and stored in the volume description.

Changing the UUID implies changing the OID of all files which have one (shortcuts, etc.) if you do not want to defeat the file tracking procedures. To me this is reasonable at formatting time only.

And using the volume UUID for "/dev/disk/by-uuid" instead of the serial number would enable tracking files on Linux.

I have the following in the boot sector of my NTFS partition:
Code:
# dd if=/dev/sda1 bs=1 count=8 skip=72 | hexdump -C
00000000  66 58 52 38 97 52 38 74                           |fXR8.R8t|
00000008

And the following in /dev/disk/by-uuid:
Code:
# ls -1 /dev/disk/by-uuid
41a06584-b407-4e66-81df-67b8f5d616e6
597e128f-ded0-434a-b7fc-e7ed652378d4
7438529738525866
a5833a95-f665-43aa-860c-61f1a523d0e5

So you can see that my distribution (openSUSE 11.4 development branch) is indeed using the 8-byte long one from the boot sector as UUID. And Ubuntu is doing the same, and in Ubuntu support forums I can read that people are using scripts which change this number: See e.g. http://www.linux.com/community/blogs/ho ... ition.html and many, many threads at http://ubuntuforums.org/ where people are actually doing that.

Good luck I asked instead of doing it. That's exactly why I ask. But I can't believe that two distributions are independently doing it wrong. Either one copied it from the other, or some widespread library is doing it wrong.


Fri Dec 10, 2010 11:37
Profile

Joined: Fri Dec 10, 2010 03:40
Posts: 6
Post Re: Feature request: Utility to fix geometry info & change UUID
Sorry, I think I have misunderstood you regarding the serial number vs. UUID. I just hear for the first time that there are two; so far I only knew about the 8-byte long one in the boot sector, which is used as a UUID in the sense of /dev/disk/by-uuid (on openSUSE and Ubuntu at least) although it isn't formally a UUID (I know that a UUID is normally 16 bytes long). What I'm proposing is a safe utility to change this one (the one which is usable and used on Linux) because there is demand for changing it and people are using dangerous "dd" commands.


Fri Dec 10, 2010 11:46
Profile

Joined: Fri Dec 10, 2010 03:40
Posts: 6
Post Re: Feature request: Utility to fix geometry info & change UUID
What I've found out so far:
  • NTFS has an 8-byte serial number that is stored in the boot sector (and backup boot sector) at offset 0x48 and can be queried under Windows with the command "fsutil fsinfo ntfsinfo C:".
  • This 8-byte serial number is used under Linux (in /dev/disk/by-uuid). There is demand for the ability to change it because people get duplicate serial numbers when they clone their filesystems.
  • Under Windows, only the least significant 4 bytes of this number are shown in directory listings (commands "fsutil fsinfo volumeinfo C:\" or simply "dir").
  • The least significant four bytes also play a role in Windows Product Activation (http://www.licenturion.com/xp/fully-licensed-wpa.txt).

What I conclude:
  • If Linux users need to change the NTFS serial number, they should do so with a specialized (yet unwritten) utility rather than random and dangerous "dd" commands.
  • The utility should of course keep boot sector and backup boot sector consistent (which the random scripts that are floating around do not).
  • The utility should offer an option to change the most significant 4 bytes and the least significant 4 bytes independently (or at least, to change the most significant 4 bytes while preserving the least significant 4 bytes). This way, Linux users could make their serial numbers unique without confusing Windows too much.


Sat Dec 11, 2010 00:37
Profile

Joined: Fri Dec 10, 2010 03:40
Posts: 6
Post Re: Feature request: Utility to fix geometry info & change UUID
Just for the records: I assume that the 16-byte UUID you mentioned above is the one that can be retrieved with the command
Code:
ntfsinfo -F \$Volume /dev/sda1 | grep "Object ID"

This one seems to be rather irrelevant to me; I didn't ask for it, and it doesn't seem to be used on Linux anywhere at this time. Filesystems created with mkntfs don't even have it!


Sat Dec 11, 2010 02:09
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi,

Quote:
I assume that the 16-byte UUID you mentioned above is the one that can be retrieved with the command
Code:
ntfsinfo -F \$Volume /dev/sda1 | grep "Object ID"

Yes
Quote:
This one seems to be rather irrelevant to me; I didn't ask for it, and it doesn't seem to be used on Linux anywhere at this time

This depends on your needs. This object_id is used to track files which have been relocated in a local network http://msdn.microsoft.com/en-us/library/aa363997(VS.85).aspx A simple example is when you have a Windows shortcut designating a file and you move the file to a different volume.

Regards

Jean-Pierre


Sat Dec 11, 2010 09:52
Profile

Joined: Thu Jan 12, 2012 13:11
Posts: 5
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi,

Could you tell me what is the status of this feature request.

I am implementing support for changing UUIDs in GParted (e.g. after copying a filesystem). For NTFS, 'UUID' translates to 'volume serial number', and as GParted makes use of native file-system tools, support for this feature in NTFS-3G would be very welcome.

Regards,

Rogier.


Thu Jan 12, 2012 13:25
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi,

Quote:
Could you tell me what is the status of this feature request.

This is one of the tools available for download in the miscellaneous tools ("tools.zip") from http://www.tuxera.com/community/ntfs-3g-advanced/

Regards

Jean-Pierre


Thu Jan 12, 2012 15:01
Profile

Joined: Thu Jan 12, 2012 13:11
Posts: 5
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi,

Thank you for your quick reply.

jpa wrote:
This is one of the tools available for download in the miscellaneous tools ("tools.zip") from http://www.tuxera.com/community/ntfs-3g-advanced/


I downloaded the archive, and I assume you are referring to the ntfsfixboot utility.

I compiled and ran it, and I checked the source code, and it seems to me that it can only fix the geometry, but it doesn't modify the volume serial number. However, I am specifically interested in the latter.

Do you have, or know of, another utility to change the volume serial number ?

Regards,

Rogier.

P.S. FYI: are you aware that the file 'COPYING' contains the GPLv2 license, implying, to the superficial observer, that all the tools are licensed GPLv2 (which most of them are), whereas the ntfsfixboot utility is actually licensed GPLv3+ ?


Thu Jan 12, 2012 15:39
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi again,

Quote:
I compiled and ran it, and I checked the source code, and it seems to me that it can only fix the geometry, but it doesn't modify the volume serial number. However, I am specifically interested in the latter.

Hmm, I am putting it into my todo list...

What exactly is your need (forcing a specific value, generating a new random one, changing partially, etc.), and in what circumstances (you mentioned copying a file system, so must/can/should this be related to ntfsclone) ?

Regards

Jean-Pierre


Thu Jan 12, 2012 16:39
Profile

Joined: Thu Jan 12, 2012 13:11
Posts: 5
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi,

Quote:
What exactly is your need (forcing a specific value, generating a new random one, changing partially, etc.), and in what circumstances (you mentioned copying a file system, so must/can/should this be related to ntfsclone) ?


My primary need, relating to the change I am writing for GParted, is the ability to set the serial number of an exsisting filesystem to a new, random value. Given the discussion in previous posts in this topic, users would certainly need the choice to change either the full serial number, or only the most significant half.

The demand to change the volume serial number arises because this number is used by several important tools on Linux to uniquely identify a filesystem. Having duplicate serial numbers can cause confusion and unexpected or even indeterminate behavior. There may be other reasons too, that I am not aware of, why people need this.
See also the post by mwk on Sat Dec 11, 2010 00:37.

Different circumstances / use-cases where the functionality would be used/needed:
- A user clones his NTFS filesystem, knowing (s)he will use it alongside the original filesystem on the same (Linux) machine.
- A user has previously cloned an NTFS filesystem, or somebody else did, and now it needs to be used alongside the original, or another copy, and some (system) tools that (s)he needs get confused by the duplicate serial numbers.
- Of course the user would still need the ability to make an identical ('forensic') clone of an NTFS filesystem.

So, although there certainly is a relation with ntfsclone, it is not a 1-1 relation. In addition, GParted currently does not allow combining the act of cloning with the choice of generating a new UUID or not. For both these reasons, I would need to be able to change only the serial number on an existing filesystem. An option to change the serial number while cloning, while probably useful in its own right, would not be of much use to me at this moment.

An option to force a specific serial number is not what I currently have a need for. I do think it might be of some use to other people though, and if there is a demand, I assume such an option could be added to GParted in the future. As it stands, most of the filesystems supported by GParted, including FAT16 and FAT32, have a command-line tool that supports forcing their UUID (equivalent in the way it is used to the serial number for NTFS and FAT), to a specific value.

In summary, I need a (command-line) tool:
1) that can set just the serial number of an existing filesystem to a random value.
2) that allows setting the full serial number, or just the most significant half.
3) that is safe to use (i.e. won't modify a non-NTFS filesystem, or (unless forced) a mounted filesystem).
4) that may be integrated with another tool (ntfsclone, or ntfslabel come to mind), provided point 1 is also satisfied
5) that may, but does not neccessarily need to, offer the option of forcing a specific value for the serial number.
6) that does everything it needs to - e.g. it wouldn't forget to update the backup boot sector.

As such a tool would be of most use if it were easily available to (most) GParted users, It would be most welcome if it could be included in the ntfs-3g package.

I hope this answers your questions.

Kind regards,

Rogier.


Thu Jan 12, 2012 18:29
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi Rogier,

Attached you will find patches to ntfsclone and ntfslabel for changing the serial number.

As I was short of option letters, I have defined option -I or --new-identity for changing the full serial number, and -i or --half-identity for only changing the upper half. I am not satisfied with these choices, and I will probably only define long options --new-serial-number and --new-half-serial-number, with no short option, unless a better name comes out. The acronym uuid cannot be used to avoid confusion with the Volume UUID used by Windows to locate files which have been moved to another volume.

When used in ntfsclone, the options only apply to cloning a volume or restoring a volume from an image.

The patches were tested with ntfs-3g_ntfsprogs-2011.10.9AR.1, and they should also be usable with ntfs-3g_ntfsprogs-2011.10.9-RC

Please test further.

Regards

Jean-Pierre


Attachments:
serial-patches.zip [4.57 KiB]
Downloaded 971 times
Fri Jan 13, 2012 11:41
Profile

Joined: Thu Jan 12, 2012 13:11
Posts: 5
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi Jean-Pierre,

jpa wrote:
Attached you will find patches to ntfsclone and ntfslabel for changing the serial number.

Thanks very much. I hadn't expected them so soon !

jpa wrote:
As I was short of option letters, I have defined option -I or --new-identity for changing the full serial number, and -i or --half-identity for only changing the upper half. I am not satisfied with these choices, and I will probably only define long options --new-serial-number and --new-half-serial-number, with no short option, unless a better name comes out. The acronym uuid cannot be used to avoid confusion with the Volume UUID used by Windows to locate files which have been moved to another volume.

I agree that 'identity' does not sound right. An option name based on the string 'serial' (or maybe 'volume-serial', e.g.: --new[-half]-volume-serial[-number]) souds perfectly sensible to me, as far as I am qualified to judge. IMHO, people using it should be reading the manual anyway.
A choice that included the string 'uuid' would have surprised me...

jpa wrote:
The patches were tested with ntfs-3g_ntfsprogs-2011.10.9AR.1, and they should also be usable with ntfs-3g_ntfsprogs-2011.10.9-RC,

I am testing with ntfs-3g_ntfsprogs-2011.10.9AR.1.

jpa wrote:
Please test further.

Feedback so far:

Could you consider changing:
Code:
srandom(time((time_t*)NULL));

in ntfslabel.c and ntfsclone.c to something like:
Code:
srandom(time((time_t*)NULL) ^ (getpid() << 16));

Or anything else that ensures that different seeds are used, and therefore different serial numbers are generated, within the same one-second timeframe:
Code:
parted:root ~ 60 # ntfslabel -I /dev/vdd10 ; ntfslabel -I /dev/vdd13
New serial number : 0x4ed66dd951961a30
Fs-10-ntfs
New serial number : 0x4ed66dd951961a30
Fs-13-ntfs


Quote:
When used in ntfsclone, the options only apply to cloning a volume or restoring a volume from an image.

When using the option -I or -i while cloning to an image file, a new serial number is still reported, which can be confusing. I have verified, and it is not stored in the image (or at least, it doesn't seem to be: a restore still results in the old serial number):
Code:
parted:root ~ 70 # ntfsclone -i -s -O /tmp/vdd13.clone /dev/vdd13
ntfsclone v2011.10.9AR.1 (libntfs-3g)
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 52424704 bytes (53 MB)
Current device size: 62914560 bytes (63 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Space in use       : 3 MB (4.9%)   
Saving NTFS to image ...
New serial number      : 0x3642475f3b7b2b27
100.00 percent completed
Syncing ...


When generating half a new serial number, ntfsclone reports a fully new serial number, but uses only the most significant part of it:
Code:
parted:root ~ 80 # blkid /dev/vdd13
/dev/vdd13: LABEL="Fs-48-ntfs" UUID="4ED66DD951961A30" TYPE="ntfs"
parted:root ~ 81 # ntfsclone -i -O /dev/vdc13 /dev/vdd13
ntfsclone v2011.10.9AR.1 (libntfs-3g)
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 52424704 bytes (53 MB)
Current device size: 62914560 bytes (63 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Space in use       : 3 MB (4.9%)   
Cloning NTFS ...
New serial number      : 0x528c9ef4144a990
100.00 percent completed
Syncing ...
parted:root ~ 82 # blkid /dev/vdc13
/dev/vdc13: LABEL="Fs-48-ntfs" UUID="0528C9EF51961A30" TYPE="ntfs"

ntfslabel does report the correct (new) serial number when changing only half of it.

So far my comments for the moment.

Please be aware that I am not able to test the effect of the new UUIDs on an MS Windows installation. I'll try to get somebody else to have a look at it as well.

Regards,

Rogier.


Fri Jan 13, 2012 15:02
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi Rogier,

Thank you for your report after testing the patches.

Quote:
An option name based on the string 'serial' (or maybe 'volume-serial', e.g.: --new[-half]-volume-serial[-number]) souds perfectly sensible to me

I have opted for --new-serial and --new-half-serial with no short option letter.
Quote:
different serial numbers are generated, within the same one-second timeframe:

Done.
Quote:
When using the option -I or -i while cloning to an image file, a new serial number is still reported, which can be confusing

Fixed.
Quote:
When generating half a new serial number, ntfsclone reports a fully new serial number, but uses only the most significant part of it:

Fixed.
Quote:
Please be aware that I am not able to test the effect of the new UUIDs on an MS Windows installation. I'll try to get somebody else to have a look at it as well.

Better save the full bootsector (or full image) before the test : Windows could decide that the clone is an illegal copy and refuse to boot.

Also fixed :
- No error report from ntfslabel when the option "--no-action" is set.
- No update (and warn) of the backup boot sector if is does not match the main boot sector

The latter may be needed because the backup boot sector may be missing. A typical case for this is after downsizing a partition : this must be done on a full size partition hence ntfsresize does not know exactly where the last sector will be. In most cases the size is given roughly (in MB rather than in sectors), so it is subject to interpretations. The user is expected to do a chkdsk after the resizing, but may want to change the serial number before.

The patches will be included in the next advanced version (in a few weeks time).

Regards

Jean-Pierre


Attachments:
serial-patches.zip [4.95 KiB]
Downloaded 988 times
Sat Jan 14, 2012 16:19
Profile

Joined: Thu Jan 12, 2012 13:11
Posts: 5
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi Jean-Pierre,

Thank you very much for the updated patch.

In the mean time, I have been implementing support for setting the volume serial number in GParted. One thing that I ran into, is that the NTFS-3G utilities currently have no way of reporting the current volume serial number, at least not that I have discovered. While GParted has a fallback, a native implementation would be preferable. Would it be possible to add an option to ntfslabel to report the volume serial number without changing it ?

I have also been thinking some more about the implications of windows product activation. Most users will not know or be able to save and restore a bootsector. At the very least, they would be hesitant to do so, not knowing what the possible side-effects could be. Nevertheless, I do think there will be a need to set / revert the volume serial number to a known value that is compatible with the windows product activation key. Could you consider adding an option that allows the new serial number to be specified, instead of generated randomly ?

Quote:
The patches will be included in the next advanced version (in a few weeks time).

That is good news !

Kind regards,

Rogier.


Sun Jan 15, 2012 22:39
Profile
NTFS-3G Lead Developer

Joined: Tue Sep 04, 2007 17:22
Posts: 1286
Post Re: Feature request: Utility to fix geometry info & change UUID
Hi Rogier,

Quote:
Would it be possible to add an option to ntfslabel to report the volume serial number without changing it ?

This is surely possible.
Quote:
I have also been thinking some more about the implications of windows product activation. Most users will not know or be able to save and restore a bootsector.

These users will probably not need to change the serial number (of their system volume).
Quote:
Could you consider adding an option that allows the new serial number to be specified, instead of generated randomly ?

I probably can.

Regards

Jean-Pierre


Mon Jan 16, 2012 22:20
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ] 


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Original forum style by Vjacheslav Trushkin.