Win32::UTCFileTime - Get/set UTC file times with stat/utime on Win32
# Override built-in stat()/lstat()/utime() within current package only: use Win32::UTCFileTime qw(:DEFAULT $ErrStr); @stats = stat $file or die "stat() failed: $ErrStr\n"; $now = time; utime $now, $now, $file; # Or, override built-in stat()/lstat()/utime() within all packages: use Win32::UTCFileTime qw(:globally); ... # Use an alternative implementation of stat() instead: use Win32::UTCFileTime qw(alt_stat $ErrStr); @stats = alt_stat($file) or die "alt_stat() failed: $ErrStr\n";
NOTE: In Perl 5.33.5, the built-in stat() and utime() functions were rewritten (and a proper implementation of lstat() was added) in such a way that the UTC file time handling is now correct (including the case of utime() being used on directories) regardless of which compiler perl is built with, thus rendering this module wholly redundant for Perl 5.33.5 or later. The remainder of this man page is written from the perspective of earlier versions of Perl.
stat()
utime()
lstat()
This module provides replacements for Perl's built-in stat() and utime() functions that respectively get and set "correct" UTC file times instead of the erroneous values read and written by Microsoft's implementation of stat(2) and utime(2), which Perl's built-in functions inherit on Win32 when built with the Microsoft C library in Visual Studio 2013 (VC12) or earlier.
stat(2)
utime(2)
The bugs in the Microsoft C library have since been fixed, so there is mostly no need for this module if you build perl with Visual Studio 2015 (VC14.0) or later, except that the use of utime() on directories is a Perl extension over the underlying Microsoft C library function and still has incorrect UTC file time handling even for perls built with VC14.0 or later.
For completeness, a replacement for Perl's built-in lstat() function is also provided, although in practice that is unimplemented on Win32 and just calls stat() anyway. (Note, however, that it calls the original stat(), not the override provided by this module, so you must use the lstat() override provided by this module if you want "correct" UTC file times from lstat().)
The problem with Microsoft's stat(2) and utime(2), and hence Perl's built-in stat(), lstat() and utime() when built with the faulty Microsoft C library, is basically this: file times reported by stat(2) or stored by utime(2) may change by an hour as we move into or out of daylight saving time (DST) if the computer is set to "Automatically adjust clock for daylight saving changes" (which is the default setting) and the file is stored on an NTFS volume (which is the preferred filesystem used by Windows NT onwards).
It seems particularly ironic that the problem should afflict the NTFS filesystem because the time_t values used by both stat(2) and utime(2) express UTC-based times, and NTFS stores file times in UTC. However, Microsoft's implementation of both of these functions use a variety of Win32 API calls that mangle the numbers in ways that do not quite turn out right when a DST season change is involved. On FAT volumes, the filesystem used by Windows 95/98/ME, file times are stored in local time and are put through even more contortions by these functions, but actually emerge correctly, so file times are stable across DST seasons on FAT volumes. The NTFS/FAT difference is taken into account by this module's replacement stat(), lstat() and utime() functions so that corrections are not erroneously applied when they should not be.
time_t
The problems that arise when mangling time values between UTC and local time are because the mapping from UTC to local time is not one-to-one. It is straightforward to convert UTC to local time, but there is an ambiguity when converting back from local time to UTC involving DST. The Win32 API provides two documented functions (FileTimeToLocalFileTime() and LocalFileTimeToFileTime()) for these conversions that resolve the ambiguity by, arguably "wrongly", using an algorithm involving the current system time rather than the file time being converted to decide whether to apply a DST correction; the advantage of this scheme is that these functions are exact inverses. Another, undocumented, function is also used internally by stat(2) for the tricky local time to UTC conversion, which, "correctly", uses the file time being converted to decide whether to apply a DST correction. The standard C library provides localtime(3) for UTC to local time conversion, albeit from time_t format to struct tm format, (and also gmtime(3) for the same structure-conversion without converting to local time), and mktime(3) for local time to UTC conversion, applying a DST correction or not as instructed by one of the fields in its struct tm argument.
FileTimeToLocalFileTime()
LocalFileTimeToFileTime()
localtime(3)
struct tm
gmtime(3)
mktime(3)
Additionally, if you build perl with Visual Studio 2013 (VC12) then perl's utime() function will suffer from a new bug introduced into the C RTL DLL's utime(2) function.
All the bugs are fixed in Visual Studio 2015 (VC14.0) onwards, but note that the C RTL DLL's utime(2) function isn't used for utime() calls on directories, which unfortunately retain their buggy behaviour.
See "BACKGROUND REFERENCE" for more details.
The replacement stat() and lstat() functions provided by this module behave identically to Perl's built-in functions of the same name, except that:
the last access time, last modification time and creation time return values are all "correct" UTC-based values, stable across DST seasons;
the argument (or $_ if no argument is given) must be a file (path or name), not a filehandle (and not the special filehandle consisting of an underscore character either).
$_
In fact, both of these replacement functions work by calling Perl's corresponding built-in function first and then overwriting the file time fields in the lists thus obtained with the corrected values. In this way, all of the extra things done by Perl's built-in functions besides simply calling the underlying C stat(2) function are inherited by this module.
(The replacement functions also incorporate one slight improvement to the built-in functions that was introduced in Perl 5.8.9 (namely, that they work correctly on directories specified with trailing slashes or backslashes), thus making this improvement available even when using an older version of Perl.)
The replacement utime() function provided by this module behaves identically to Perl's built-in function of the same name, except that:
the last access time and last modification time arguments are both "correctly" interpreted as UTC-based values, stable across DST seasons;
no warnings about "Use of uninitialized value in utime" are produced when either file time argument is specified as undef.
undef
In particular, the one extra thing done by Perl's built-in function besides simply calling the underlying C utime(2) function (namely, providing a fix so that it works on directories as well as files) is also incorporated into this module's replacement function.
All three replacement functions are exported to the caller's package by default. A special :globally export pseudo-symbol is also provided that will export all three functions to the CORE::GLOBAL package, which effectively overrides the Perl built-in functions in all packages, not just the caller's. The alt_stat() function is only exported when explicitly requested.
:globally
alt_stat()
stat([$file])
Gets the status information for the file $file. If $file is omitted then $_ is used instead.
In list context, returns the same 13-element list as Perl's built-in stat() function on success, or returns an empty list and sets $ErrStr on failure.
For convenience, here are the members of that 13-element list and their meanings on Win32:
0 dev drive number of the disk containing the file (same as rdev) 1 ino not meaningful on Win32; always returned as 0 2 mode file mode (type and permissions) 3 nlink number of (hard) links to the file; always 1 on non-NTFS drives 4 uid numeric user ID of file's owner; always 0 on Win32 5 gid numeric group ID of file's owner; always 0 on Win32 6 rdev drive number of the disk containing the file (same as dev) 7 size total size of file, in bytes 8 atime last access time in seconds since the epoch 9 mtime last modification time in seconds since the epoch 10 ctime creation time in seconds since the epoch 11 blksize not implemented on Win32; returned as '' 12 blocks not implemented on Win32; returned as ''
where the epoch was at 00:00:00 Jan 01 1970 UTC and the drive number of the disk is 0 for A:, 1 for B:, 2 for C: and so on.
Because the mode contains both the file type (the S_IFDIR bit is set if $file specifies a directory; the S_IFREG bit is set if the $file specifies a regular file) and its permissions (the user read/write bits are set according to the file's permission mode; the user execute bits are set according to the filename extension), you should mask off the file type portion and printf() using a "%04o" if you want to see the real permissions:
S_IFDIR
S_IFREG
printf()
"%04o"
$mode = (stat($filename))[2]; printf "Permissions are %04o\n", $mode & 07777;
You can also import symbolic mode constants (S_IF*) and functions (S_IS*()) from the Fcntl module to assist in examining the mode. See "stat" in perlfunc for more details.
S_IF*
S_IS*()
Note that you cannot use this module in conjunction with the File::stat module (which provides a convenient, by-name, access mechanism to the fields of the 13-element list) because both modules operate by overriding Perl's built-in stat() function. Only the second override to be applied would have effect.
In scalar context, returns a Boolean value indicating success or failure (and sets $ErrStr on failure).
lstat([$link])
Gets the status information for the symbolic link $link. If $file is omitted then $_ is used instead. This is the same as stat() on Win32, which does not implement symbolic links.
alt_stat([$file])
Behaves almost identically to stat() above, but uses this module's own implementation of the standard C library stat(2) function that can succeed in some cases where Microsoft's implementation fails.
Microsoft's stat(2), and hence Perl's built-in stat() and the replacement stat() function above, calls the Win32 API function FindFirstFile(). That function is used to search a directory for a file, and thus requires the process to have "List Folder Contents" permission on the directory containing the $file in question. If that permission is denied then stat() will fail.
FindFirstFile()
alt_stat() avoids this problem by using a different Win32 API function, CreateFile(), instead. That function opens a file directly and hence does not require the process to have "List Folder Contents" permission on the parent directory. However, on Windows 95 platforms, it cannot open directories at all and will only set the st_mode field correctly; the other fields will be set to zero, like the Perl built-in stat() and lstat() functions do for directories on which stat(2) fails (which can also happen in that case, e.g. on sharenames).
CreateFile()
st_mode
The main disadvantage with using this function is that the entire struct stat has to be built by hand by it, rather than simply inheriting most of it from the Microsoft stat(2) call and then overwriting the file time fields. Thus, some of the fields, notably the st_mode field, which is somewhat ambiguous on Win32, may have different values to those that would have been set by the other stat() functions.
struct stat
utime($atime, $mtime, @files)
Sets the last access time and last modification time to the values specified by $atime and $mtime respectively for each of the files in @files. The process must have write access to each of the files concerned in order to change these file times.
Returns the number of files successfully changed and sets $ErrStr if one or more files could not be changed.
The times should both be specified as the number of seconds since the epoch, where the epoch was at 00:00:00 Jan 01 1970 UTC. If the undefined value is used for either file time argument then the current time will be used for that value.
Note that the 11th element of the 13-element list returned by stat() is the creation time on Win32, not the inode change time as it is on many other operating systems. Therefore, neither Perl's built-in utime() function nor this replacement function set that value to the current time as would happen on other operating systems.
Last error message.
If any function fails then a description of the last error will be set in this variable for use in reporting the cause of the failure, much like the use of the Perl Special Variables $! and $^E after failed system calls and Win32 API calls. Note that $! and/or $^E may also be set on failure, but this is not always the case so it is better to check $ErrStr instead. Any relevant messages from $! or $^E will form part of the message in $ErrStr anyway. See "Error Values" for a listing of the possible values of $ErrStr.
$!
$^E
If a function succeeds then this variable will be set to the null string.
Control whether to try alt_stat() if CORE::stat() or CORE::lstat() fails.
CORE::stat()
CORE::lstat()
Boolean value.
As documented in the "DESCRIPTION" section above, the replacement stat() and lstat() functions each call their built-in counterparts first and then overwrite the file time fields in the lists thus obtained with the corrected values. Setting this variable to a true value will cause the replacement functions to switch to alt_stat() (via a goto &NAME call) if the CORE::stat() or CORE::lstat() call fails.
goto &NAME
The default value is 0, i.e. the alt_stat() function is not tried.
This module may produce the following diagnostic messages. They are classified as follows (a la perldiag):
(W) A warning (optional). (F) A fatal error (trappable). (I) An internal error that you should never see (trappable).
(W) The specified file descriptor for the specified file could not be closed after updating the file times using it. The system error message corresponding to the standard C library errno variable is also given.
errno
(W) The specified file object handle for the specified file could not be closed after reading file information from it. The system error message corresponding to the Win32 API last error code is also given.
(W) The specified file object handle for the specified file could not be closed after updating the file times using it. The system error message corresponding to the Win32 API last error code is also given.
(I) The SYSTEMTIME representation of the epoch of time_t values (namely, 00:00:00 Jan 01 1970 UTC) could not be converted to its FILETIME representation. The system error message corresponding to the Win32 API last error code is also given.
SYSTEMTIME
FILETIME
(W) The operating system platform (i.e. Win32s, Windows (95/98/ME), Windows NT or Windows CE) could not be determined. This information is used by the alt_stat() function to decide whether a ".cmd" file extension represents an "executable file" when setting up the st_mode field of the struct stat. A Windows NT platform is assumed in this case. The system error message corresponding to the Win32 API last error code is also given.
(F) You attempted to lookup the value of the specified constant in the Win32::UTCFileTime module, but that constant is unknown to this module.
(W) The number of hard links to the specified file is greater than the largest short int, and therefore cannot be assigned to the st_nlink field of the struct stat set-up by alt_stat(). The largest short int itself is used instead in this case.
short int
st_nlink
(I) There was an unexpected error looking up the value of the specified constant: the constant-lookup function itself is apparently not defined.
(I) There was an unexpected error looking up the value of the specified constant: the C component of the constant-lookup function returned an unknown type.
(I) You attempted to lookup the value of the specified constant in the Win32::UTCFileTime module, but that constant is apparently not defined.
Each function sets $ErrStr to a value indicating the cause of the error when they fail. The possible values are as follows:
File information could not be read from an open file handle on the specified file. The system error message corresponding to the Win32 API last error code is also given.
A file object handle could not be obtained for the specified file descriptor, which the specified file has been opened as for updating. The system error message corresponding to the standard C library errno variable is also given.
The specified file could not be opened for reading the file information. The system error message corresponding to the Win32 API last error code is also given.
The specified file could not be opened for updating the file times. The system error message corresponding to the Win32 API last error code is also given.
The file times could not be updated on an open file handle on the specified file. The system error message corresponding to the Win32 API last error code is also given.
The CORE::stat() or CORE::lstat() function failed for the specified file or link. (The replacement stat() and lstat() functions call their CORE counterparts before getting the "correct" UTC file times.) The system error message corresponding to the standard C library errno variable is also given.
The specified filename passed to the alt_stat() function contains a DOS-style wildcard character, namely "?" or "*". The function is not able to handle filenames of this format.
In some cases, the functions may also leave the Perl Special Variables $! and/or $^E set to values indicating the cause of the error when they fail; one or the other of these will be incorporated into the $ErrStr message in such cases, as indicated above. The possible values of each are as follows ($! shown first, $^E underneath):
EACCES
ERROR_ACCESS_DENIED
[utime() only.] One or more of the @files is read-only. (The process must have write access to each file to be able to update its file times.)
EMFILE
ERROR_TOO_MANY_OPEN_FILES
The maximum number of file descriptors has been reached. (Each file must be opened in order to read file information from it or to update its file times.)
ENOENT
ERROR_FILE_NOT_FOUND
The filename or path in $file was not found.
Note that since each function uses Win32 API functions rather than standard C library functions, they will probably only set $^E (which represents the Win32 API last error code, as returned by GetLastError()), not $! (which represents the standard C library errno variable).
GetLastError()
Other values may also be produced by various functions that are used within this module whose possible error codes are not documented.
See $!, %!, $^E and Error Indicators in perlvar, Win32::GetLastError() and Win32::FormatMessage() in Win32, and Errno and Win32::WinError for details on how to check these values.
%!
Win32::GetLastError()
Win32::FormatMessage()
A number of Microsoft Knowledge Base articles refer to the odd characteristics of the Win32 API functions and the Microsoft C library functions involved, in particular see:
128126: FileTimeToLocalFileTime() Adjusts for Daylight Saving Time
129574: Time Stamp Changes with Daylight Savings
158588: Obtaining Universal Coordinated Time (UTC) from NTFS Files
190315: Some CRT File Functions Adjust For Daylight Savings Time
As these articles themselves emphasize, the behaviour in question is by design, not a bug. As an aside, another Microsoft Knowledge Base article (214661: FIX: Daylight Savings Time Bug in C Run-Time Library) refers to a different problem involving the Microsoft C library that was confirmed as a bug and was fixed in Visual C++ 6.0 Service Pack 3, so it is worth ensuring that your edition of Visual C++ is upgraded to at least that Service Pack level when you build perl and this module. (At the time of writing, Service Pack 6 is the latest available for Visual C++ 6.0.)
An excellent overview of the problem with Microsoft's stat(2) was written by Jonathan M Gilligan and posted on the Code Project website (https://www.codeproject.com/). He has kindly granted permission to use his article here to describe the problem more fully. A slightly edited version of it now follows; the original article can be found at the URL https://www.codeproject.com/datetime/dstbugs.asp.
(The article was accompanied by a C library, adapted from code written for CVSNT (http://www.cvsnt.org/) by Jonathan and Tony M Hoyle, which implemented the solution outlined at the end of his article. The solution provided by this module is partly based on that library and the original CVSNT code itself (version 2.0.4), which both authors kindly granted permission to use under the terms of the Perl Artistic License as well as the GNU GPL.)
Not many Windows developers seem aware of it, but Microsoft deliberately designed Windows NT to report incorrect file creation, modification, and access times. This decision is documented in the Knowledge Base in articles Q128126 and Q158588. For most purposes, this behaviour is innocuous, but as Microsoft writes in Q158588,
After the automatic correction for Daylight Savings Time, monitoring programs comparing current time/date stamps to reference data that were not written using Win32 API calls which directly obtain/adjust to Universal Coordinated Time (UTC) will erroneously report time/date changes on files. Programs affected by this issue may include version-control software, database-synchronization software, software-distribution packages, backup software...
This behaviour is responsible for a flood of questions to the various support lists for CVS, following the first Sunday in April and the last Sunday in October, with scores of people complaining that CVS now reports erroneously that their files have been modified. This is commonly known as the "red file bug" because the WinCVS shell uses red icons to indicate modified files.
Over the past two years, several people have made concerted efforts to fix this bug and determine the correct file modification times for files both on NTFS and FAT volumes. It has proved surprisingly difficult to solve this problem correctly. I believe that I have finally gotten everything right and would like to share my solution with anyone else who cares about this issue.
Run the following batch file on a computer where C: is an NTFS volume and A: is a FAT-formatted floppy disk. You will need write access to C:\ and A:\. This script will change your system time and date, so be prepared to manually restore them afterwards.
REM Test_DST_Bug.bat REM File Modification Time Test Date /T Time /T Date 10/27/2001 Time 10:00 AM Echo Foo > A:\Foo.txt Time 10:30 AM Echo Foo > C:\Bar.txt dir A:\Foo.txt C:\Bar.txt Date 10/28/2001 dir A:\Foo.txt C:\Bar.txt REM Prompt the user to reset the date and time. date time
The result looks something like this (abridged to save space)
C:\>Date 10/27/2001 C:\>dir A:\Foo.txt C:\Bar.txt Directory of A:\ 10/27/01 10:00a 6 Foo.txt Directory of C:\ 10/27/01 10:30a 6 Bar.txt C:\>Date 10/28/2001 C:\>dir A:\Foo.txt C:\Bar.txt Directory of A:\ 10/27/01 10:00a 6 Foo.txt Directory of C:\ 10/27/01 09:30a 6 Bar.txt
On 27 October, Windows correctly reports that Bar.txt was modified half an hour after Foo.txt, but the next day, Windows has changed its mind and decided that actually, Bar.txt was modified half an hour before Foo.txt. A naive programmer might think this was a bug, but as Microsoft emphasized, this is how they want Windows to behave.
The origin of this file time problem lies in the early days of MS-DOS and PC-DOS. Unix and other operating systems designed for continuous use and network communications have long tended to store times in GMT (later UTC) format so computers in different time zones can accurately determine the order of different events. However, when Microsoft adapted DOS for the IBM PC, the personal computer was not envisioned in the context of wide-area networks, where it would be important to compare the modification times of files on the PC with those on another computer in another time zone.
In the interest of efficiently using the very limited resources of the computer, Microsoft wisely decided not to waste bits or processor cycles worrying about time zones. To put this decision in context, recall that the first two generations of PCs did not have battery-backed real-time clocks, so you would generally put DATE and TIME commands into your AUTOEXEC.BAT file to prompt you to enter the date and time manually when the computer booted.
DATE
TIME
By the time of WinNT, wide-area networks and had become sufficiently common that Microsoft realized that the OS should measure time in some universal format that would allow different computers to compare the order (and separation) of events irrespective of their particular time zones. Although the details vary (different time structures measure time relative to different events), the net effect is that all times used internally in Win32 measure time with respect to UTC (what used to be called GMT).
Having once worked down the hall from the master atomic clock array for the United States at the National Institute of Standards and Technology in Boulder, I feel obligated to add a few words about time and systems for reporting time. Long ago, we used to refer time to GMT, or Greenwich Mean Time, which was kept by the Royal Observatory in Greenwich, England and was ultimately referred to the position of the sun as measured by the observatory. When atomic clocks became the standard for timekeeping, a new standard, called UTC emerged. UTC is a bastard acronym. In English, it stands for "Coordinated Universal Time," while in French it stands for "le temps universel coordonne." Rather than using either CUT or TUC, the nonsense compromise acronym UTC was adopted.
To understand UTC, we must first understand the more abstract International Atomic Time (TAI, le temps atomique international), which measures the number of seconds that have elapsed since approximately 1 Jan 1958, as measured by caesium atomic clocks. The second is defined to be the amount of time required for 9 192 631 770 cycles of the caesium hyperfine frequency. However, neither the day nor the year are exact multiples of this number, so we take TAI and correct it so that it corresponds to the actual motion of the earth by adding corrections such as "leap seconds." TAI measures raw atomic time. UTC measures time coordinated to the motion of the earth (i.e., so we don't end up having midnight while the sun is shining or January in midsummer). Details of what UTC really means, together with a more detailed history of timekeeping, can be found at http://ecco.bsee.swin.edu.au/chronos/GMT-explained.html.
So, what does this all have to do with file modification times on Windows computers? Windows is stuck with some serious problems integrating FAT and NTFS files compatibly. FAT records file modification times with respect to the local time zone, while NTFS records file modification (as well as creation and access times, which FAT does not record) in UTC. The first question you may want to ask is, "How should Windows report these file times?" Clearly, it would be stupid for dir and Windows Explorer to report FAT file times in the local time zone and NTFS file times in UTC. If inconsistent formats were used, users would have great difficulty determining which of two files was more recent. We must thus choose to translate one of the two file time formats when we report to the user. Most users are likely to want to know the file modification time in their local time zone. This keeps things consistent with what people learned to expect under DOS and Win16. It also is more useful to most users, who may want to know how long ago they modified a file without looking up the offset of their local time zone from UTC.
dir
It is straightforward to translate UTC to local time. You look up the offset, in minutes, between the local time zone and UTC, determine whether daylight saving is in effect and add either the standard or the daylight offset to the UTC time. However, daylight time throws a subtle wrench in the works if we try to go backwards...
If you want to translate a time in your local time zone into UTC, it seems a straightforward matter of determining whether daylight time is in effect locally and then subtracting either the standard or the daylight offset from the local time to arrive at UTC. A subtle problem emerges due to the fact that the mapping from UTC to local time is not one-to-one. Specifically, when we leave daylight saving time and set our clocks back, there are two distinct hour-long intervals of UTC time that map onto the same hour-long interval of local time. Consider the concrete case of 01:30 on the last Sunday in October. Let's suppose the local time zone is US Central Time (-6 hours offset from UTC when daylight time is not in effect, -5 hours when it is). At 06:00 UTC on Sunday 28 October 2001, the time in the US Central zone will be 01:00 and daylight time will be in effect. At 06:30 UTC, it will be 01:30 local. At 07:00 UTC, it will be 01:00 local and daylight time will not be in effect. At 07:30 UTC, it will be 01:30 local. Thus, for all times 01:00 <= t < 02:00 local, there will be two distinct UTC times that correspond to the given local time. This degenerate mapping means that we can't be sure which UTC time corresponds to 01:30 local time. If a FAT file is marked as having been modified at 01:30 on Oct 28 2001, we can't determine the UTC time.
When translating local file times to UTC and vice-versa, Microsoft made a strange decision. We would like to have the following code produce out_time equal to in_time
out_time
in_time
FILETIME in_time, local_time, out_time; // Assign in_time, then do this... FileTimeToLocalFileTime(&in_time, &local_time); LocalFileTimeToFileTime(&local_time, &out_time );
The problem is that if the local time zone is US Central (UTC-6 hours for standard time, UTC-5 hours for daylight time) then in_time = 06:30 Oct 28 2001 and in_time = 07:30 Oct 28 2001 both map onto the same local time, 01:30 Oct 28 2001 and we don't know which branch to choose when we execute LocalFileTimeToFileTime(). Microsoft picked an incorrect, but unambiguously invertible algorithm: move all times up an hour when daylight time is in effect on the local computer, irrespective of the DST state of the time being converted. Thus, if DST is in effect on my local computer, FileTimeToLocalFileTime() converts 06:30 Oct 28 2001 UTC to 01:30 CDT and 07:30 Oct 28 2001 UTC to 02:30 CDT. If I call the same function with the same arguments, but when DST is not in effect on my local computer, FileTimeToLocalFileTime() will convert 06:30 UTC to 00:30 CDT and 07:30 UTC to 01:30 CDT.
It may seem strange that this would affect the C library call stat(2), which allegedly returns the UTC modification time of a file. If you examine the source code for Microsoft's C library, you find that it gets the modification time thus:
// Pseudo-code listing WIN32_FIND_DATA find_buf; HANDLE hFile; FILETIME local_ft; SYSTEMTIME local_st; time_t mod_time; // FindFirstFile() returns times in UTC. // For NTFS files, it just returns the modification time stored on the disk. // For FAT files, it converts the modification time from local (which is // stored on the disk) to UTC using LocalFileTimeToFileTime(). hFile = FindFirstFile(file_name, &find_buf); // Convert UTC time to local time. FileTimeToLocalFileTime(&find_buf.ftLastWriteTime, &local_ft); FileTimeToSystemTime(&local_ft, &local_st); // Now use a private, undocumented function to convert local time to UTC // time according to the DST settings appropriate to the time being // converted! mod_time = __loctotime_t(local_st);
For a FAT file, the conversions work like this:
Raw file modification time (stored as local time) is converted to UTC by LocalFileTimeToFileTime().
UTC is converted back to local time by FileTimeToLocalFileTime(). Note that this exactly reverses the effect of the previous step, so we are left with the correct local modification time.
Local time is converted to "correct" UTC by private function.
For an NTFS file, the conversions work like this:
Raw file modification time is already stored in UTC, so we don't need to convert it.
UTC is converted to local time by FileTimeToLocalFileTime(). This applies a DST correction according to the DST setting of the current system time, irrespective of the DST setting at the file modification time.
Local time is converted to "correct" UTC by private function. Note that this does not necessarily reverse the effect of the previous step because in this step we use the DST setting of the file modification time, not the current system time.
This explains the problem I showed at the top of this article: The time reported by dir for a file on an NTFS volume changes by an hour as we move into or out of daylight saving time, despite the fact that the file hasn't been touched. FAT modification times are stable across DST seasons.
There are 3 possible ways I can think of where this inconsistency in reporting file times may cause problems:
You may be comparing a file on an NTFS volume with a time_t value stored in a file (or memory). This is frequently seen in CVS and leads to the infamous "red file bug" on the first Sunday in April and the last Sunday in October.
You may be comparing a file on a FAT volume with a time_t value.
You may be comparing a file on a FAT volume with a file on an NTFS volume.
For the first case, it's simple. Get the file times using the Win32 API call GetFileTime() instead of using the C library stat(2), and convert the FILETIME to time_t by subtracting the origin (00:00:00 Jan 01 1970 UTC) and dividing by 10,000,000 to convert "clunks" (100-nanosecond intervals) to seconds.
GetFileTime()
For the second case, stat(2) will work and return a time_t that you can compare to the stored one. If you must use GetFileTime() do not use LocalFileTimeToFileTime(). This function will apply the daylight status of the current system time, not the daylight status of the file time in the argument. Fortunately, the C library mktime(3) function will correctly convert the time if you correctly set the tm_isdst field of the struct tm.
tm_isdst
There is a bit of a chicken-and-egg problem here. Windows does not supply a good API call to let you determine whether DST was in effect at a given time. Fortunately for residents of the US and other countries that use the same logic (Daylight time starts at 2:00 AM on the first Sunday in April and ends at 2:00 AM on the last Sunday in October), you can set tm_isdst to a negative number and mktime(3) will automatically determine whether daylight time applies or not. If the file was modified in the window 1:00-2:00 AM on the last Sunday in October, it's ambiguous how mktime(3) computes the modification time.
People in time zones that do not follow the usual US daylight rule must brute-force the daylight time problem by retrieving the applicable TIME_ZONE_INFORMATION structure with GetTimeZoneInformation() and manually calculating whether daylight time applies.
TIME_ZONE_INFORMATION
GetTimeZoneInformation()
For the third case, the best bet is to follow the instructions for the second case above and compare the resultant UTC time_t with the time for the NTFS file determined in the first case above.
The library implements this solution with checking for the filesystem the file is stored under.
That is the end of Jonathan M Gilligan's article. It should be noted that although the last section, "Solutions", refers to the Win32 API function GetFileTime(), both his library and the CVSNT code that it was adapted from use a different Win32 API function instead, namely, FindFirstFile(). As seen in the pseudo-code listing above, that is the function used in Microsoft's implementation of stat(2) itself, and evidently has one advantage over GetFileTime() that is documented in the Microsoft Knowledge Base article 128126 cited above: GetFileTime() gets cached UTC times from FAT, whereas FindFirstFile() always reads the time from the file. This means that the value returned by GetFileTime() may be incorrect under FAT after a DST season change.
That should have been the end of the discussion about the problems with Microsoft's implementation of stat(2) and how to overcome them, and, indeed, versions of this module before version 1.45 did simply implement one of the solutions above (together with another enhancement to Jonathan's library, taken from the CVSNT code, to apply the correct daylight saving time rule, rather than assuming the United States' rule).
However, it came to my attention those older versions of this module did not correctly handle file times on FAT due to a further subtlety in the Win32 API function FindFirstFile(). Recall from above that on FAT, FindFirstFile() has to convert the local time stored on the disk to UTC returned by the function, and that this conversion was believed to be performed incorrectly when the file time concerned is in the opposite DST season to the current system time.
This is easily seen by performing a test similar to that at start of Jonathan's article above ("An example of the problem"). Create a file called file, change the system time to the opposite DST season and then run the program below to print out the file's last modification time:
// test1.c #include <windows.h> #include <stdio.h> void main(void) { HANDLE hndl; WIN32_FIND_DATA wfd; SYSTEMTIME st; if ((hndl = FindFirstFile("file", &wfd)) != INVALID_HANDLE_VALUE && FileTimeToSystemTime(&wfd.ftLastWriteTime, &st)) { printf("Time: %02d/%02d/%04d %02d:%02d:%02d\n", st.wDay, st.wMonth, st.wYear, st.wHour, st.wMinute, st.wSecond); CloseHandle(hndl); } }
If this is done on an NTFS drive then the file time displayed will be the correct UTC time at which the file was created (last modified). If it is done on a FAT drive then the file time displayed will be wrong by one hour.
That is fine, but notice that in the above sequence of events, the file time is only ever set to the current system time. The subtlety referred to above is only seen when the file time was itself set from the opposite DST season. This is only possible programmatically; for example, the program below uses the Win32 API function SetFileTime() to set the file time to 12:00 Jun 01 2005 UTC:
SetFileTime()
// test2.c #include <windows.h> #include <stdio.h> void main(void) { SYSTEMTIME st = { 2005, 6, 0, 1, 12, 0, 0, 0 }; FILETIME ft; HANDLE hndl; if (SystemTimeToFileTime(&st, &ft) && (hndl = CreateFile("file", GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_DELETE, NULL, OPEN_EXISTING, 0, NULL)) != INVALID_HANDLE_VALUE && SetFileTime(hndl, NULL, &ft, &ft)) { printf("File time has been set to 01/06/2005 12:00:00\n"); CloseHandle(hndl); } }
Using a program such as the one above (and varying the file time that it sets as necessary) a few simple tests show that the problem with FindFirstFile() on a FAT drive actually seems to be this: the file time retrieved is wrong by one hour if it was set in the opposite DST season to the current system time.
This, of course, includes the case where (as in <"An example of the problem">) a file is created in one DST season and is then read from the opposite DST season (for example, create a file in summer (which effectively sets a summer file time from summer) and then read it from winter), but it also includes hitherto unseen cases in which FindFirstFile() will even return the wrong time for a file time in the same DST season as the current system time if the file time was set from the opposite DST season to the current system time (for example, set a summer file time from winter and then read it from summer). Also note that, in other thus-far unseen cases, FindFirstFile() can return the correct time for a file time in the opposite DST season to the current system time if the file time was set from the same DST season as the current system time (for example, set a winter file time from summer and then read it from summer).
Thus, we appear to be concerned with when the file time was set relative to the current system time, rather than what the file time is relative to the current system time. This misunderstanding accounts for some of the test failures that versions of this module before version 1.45 exhibited on a FAT drive.
Of course, when a given file time was set is not something that can be known at a later date because this information is not stored anywhere, which seems to make a solution based on FindFirstFile() impossible. This leads us to look to other Win32 API functions for a solution instead.
GetFileInformationByHandle() was already being used in older versions of this module in two places: in the replacement stat() function as a fallback in the cases (such as a directory specified with a trailing backslash) where FindFirstFile() fails, and in the alternative alt_stat() function. It was assumed GetFileInformationByHandle() performed in the same manner as FindFirstFile() with respect to how it retrieved and/or converted file times on FAT and NTFS drives, and hence the same "corrections" were applied. However, further tests such as above with the program below shows that this is not true.
GetFileInformationByHandle()
// test3.c #include <windows.h> #include <stdio.h> void main(void) { HANDLE hndl; BY_HANDLE_FILE_INFORMATION bhfi; SYSTEMTIME st; if ((hndl = CreateFile("file", GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL)) != INVALID_HANDLE_VALUE && GetFileInformationByHandle(hndl, &bhfi) && FileTimeToSystemTime(&bhfi.ftLastWriteTime, &st)) { printf("Time: %02d/%02d/%04d %02d:%02d:%02d\n", st.wDay, st.wMonth, st.wYear, st.wHour, st.wMinute, st.wSecond); CloseHandle(hndl); } }
Comparing the output of test1.c and test3.c for a test file with its file time in each permutation of being set from and being in different DST seasons in turn, on both FAT and NTFS drives, it appears that whilst test1.c exhibits the strange behaviour described above on FAT and only returns correct UTC time reliably on NTFS, test3.c always returns correct UTC time on both FAT and NTFS.
Thus, the "corrections" that older versions of this module applied to the file times retrieved by GetFileInformationByHandle() should not have been applied, which accounts for more of the test failures that those versions exhibited on FAT drives.
The final solution is thus alarmingly simple, and much simpler than that implemented in the library accompanying Jonathan M Gilligan's article and in the related CVSNT code: simply use GetFileInformationByHandle() to retrieve the last modification time. It is always correct UTC time on both FAT and NTFS.
Another look at the source for Microsoft's C library shows that everything written above regarding the last modification time of a file is also true of the last access time and creation time, and simple modifications to the test programs above also show that GetFileInformationByHandle() always returns them correctly too, so this module corrects those values as well in the same way.
(Incidentally, the source code of Microsoft's implementation of stat(2) can be found in C:\Program Files\Microsoft Visual Studio\VC98\CRT\SRC\STAT.C if you installed Visual C++ 6.0 in its default location and selected the "VC++ Runtime Libraries -> CRT Source Code" option when installing.)
We have been looking at stat(2) in the context of getting various times associated with files, so it is natural to wonder whether the complementary function in this regard, namely utime(2), which sets file times, is afflicted in a similar way.
The answer, unfortunately, is yes. A look at the source code of Microsoft's implementation of utime(2) shows that it puts the supplied last access time and last modification time through similar, though slightly simpler, contortions to those in stat(2) before storing them in the filesystem. Briefly, it goes something like the following:
// Pseudo-code listing time_t mod_time; struct tm *local_tmb; SYSTEMTIME local_st; FILETIME local_ft; FILETIME utc_ft; // Convert UTC time to local time. local_tmb = localtime(mod_time) // Convert local time to UTC time. // (Construct SYSTEMTIME from struct tm first.) SystemTimeToFileTime(&local_st, &local_ft); LocalFileTimeToFileTime(&local_ft, utc_ft) // Set UTC time. SetFileTime(hFile, utc_ft)
In versions of this module before version 1.45 it was believed that SetFileTime() called FileTimeToLocalFileTime() on FAT drives, thus cancelling out the effect of the LocalFileTimeToFileTime() call and yielding the correct result but in fact this is not the case, which accounts for the remaining test failures that those versions exhibited on FAT drives.
Performing some tests using the following program to set various file times from the same or opposite DST season as the file time being set and then reading back the file times that were actually using test3.c above (which is believed always to return the correct file time), we can observe that test4.c always sets the file time wrongly (by one hour) if the file time being set is in the opposite DST season to the current system time, and it always sets the file time correctly if the file time being set is in the same DST season to the current system time. This is true on both FAT and NTFS drives.
// test4.c #include <windows.h> #include <stdio.h> #include <time.h> #include <sys/utime.h> static BOOL _FileTimeToUnixTime(const FILETIME *ft, time_t *ut); void main(void) { SYSTEMTIME st = { 2005, 6, 0, 1, 12, 0, 0, 0 }; FILETIME ft; time_t ut; struct utimbuf utb; if (SystemTimeToFileTime(&st, &ft) && _FileTimeToUnixTime(&ft, &ut) && (utb.actime = utb.modtime = ut) && utime("file", &utb) != -1) { printf("File time has been set to 01/06/2005 12:00:00\n"); } } static BOOL _FileTimeToUnixTime(const FILETIME *ft, time_t *ut) { SYSTEMTIME basest = { 1970, 1, 0, 1, 0, 0, 0, 0 }; FILETIME baseft; ULARGE_INTEGER it; ULONGLONG clunks_per_second = 10000000L; if (!SystemTimeToFileTime(&basest, &baseft)) return FALSE; it.QuadPart = ((ULARGE_INTEGER *)ft)->QuadPart; it.QuadPart -= ((ULARGE_INTEGER *)&baseft)->QuadPart; it.QuadPart /= clunks_per_second; *ut = it.LowPart; return TRUE; }
The reason for this behaviour is simply that the localtime(3) call in Microsoft's implementation of utime(2) shown above converts the time correctly, while the later LocalFileTimeToFileTime() call does not. The final <SetFileTime()> call then behaves correctly on both FAT and NTFS, just as GetFileInformationByHandle() was found to do further above.
We therefore have a situation slightly simpler than that for stat(2): under both FAT and NTFS, two conversions are applied that are not the exact inverses of each other, leaving the UTC file times supplied potentially wrong by one hour.
Thus, the time set by utime(2) for a file on either a FAT or an NTFS volume is incorrect by an hour when the time being set is in a different DST season to the current system time.
The solution to this problem implemented by this module is as simple as that to the stat(2) problem: just use the appropriate Win32 API function instead (in this case, SetFileTime()).
This whole sorry saga still has one more sting in the tail. In Visual Studio 2013 (VC12) some changes were made to the stat(2) function to resolve the subtle DST bug in it, but the utime(2) function did not have corresponding changes made to it so that calling the utime(2) function to set the specified file access and modification times and then calling the stat(2) function to retrieve those times can yield different times than were specified.
Specifically, the times returned by stat(2) can be one hour different to the times passed to utime(2) depending on DST values. The roundtrip fails for dates and times with the opposite DST value to the computer's current date and time, which was not the case in previous versions of Visual Studio since stat(2) used to have the same DST bug as utime(2), thus at least yielding consistent results.
Details of this problem could previously be found here: https://connect.microsoft.com/VisualStudio/feedback/details/811534/utime-sometimes-fails-to-set-the-correct-file-times-in-visual-c-2013. Unfortunately, Microsoft Connect has now been retired. Visual C++ bugs should have been migrated over to the Developer Community site for C++ (see: https://developercommunity.visualstudio.com/spaces/62/index.html), but this particular bug doesn't seem to have been migrated, perhaps because it is now resolved (see "A Happy Ending" below).
Fortunately, the replacement utime() function provided by this module fixes this problem too.
With the release of Visual Studio 2015 (VC14.0), Microsoft finally resolved all the problems with stat(2) and utime(2), rendering this module mostly redundant (but see below). The following words are taken from "Microsoft C/C++ change history 2003 - 2015" (see: https://docs.microsoft.com/en-us/cpp/porting/visual-cpp-change-history-2003-2015?view=msvc-160#VC_2015):
Visual Studio 2015 Conformance Changes C Runtime Library (CRT) In previous versions, the _stat, fstat, and _utime functions handle daylight savings time incorrectly. Prior to Visual Studio 2013, all of these functions incorrectly adjusted standard time times as if they were in daylight time. In Visual Studio 2013, the problem was fixed in the _stat family of functions, but the similar problems in the fstat and _utime families of functions were not fixed. This partial fix led to problems due to the inconsistency between the functions. The fstat and _utime families of functions have now been fixed, so all of these functions now handle daylight savings time correctly and consistently.
There is still one fly in the ointment: The Microsoft implementation of utime(2) doesn't work on directories, and whilst Perl's built-in utime() function works around that, it unfortunately does so in such a way that the UTC file time handling is still not correct even for perls built with VC14.0 or later.
However, all this was finally put to bed with the resolution of CPAN RT#18513 (see https://github.com/Perl/perl5/issues/6080) in Perl 5.33.5, in which Perl's built-in stat() and utime() functions were rewritten (and a proper implementation of lstat() was added) in such a way that the UTC file time handling is now correct (including the case of utime() being used on directories) regardless of which compiler perl is built with, thus rendering this module wholly redundant.
The following symbols are, or can be, exported by this module:
stat, lstat, utime.
stat
lstat
utime
alt_stat, $ErrStr, :globally.
alt_stat
$ErrStr
None.
Any of the Win32 API functions involved here, or indeed Microsoft's implementation of stat(2) and utime(2) themselves, could, of course, change in future Windows operating systems. Such changes could render inappropriate the corrections being applied by this module. (In fact, the Microsoft Knowledge Base article Q158588 cited above specifically mentions that the behaviour of GetFileTime() under FAT may be changed to match the behaviour under NTFS in a future version of Windows NT. That particular change, however, would have no effect on this module because, as mentioned above, GetFileTime() is not used by it.)
Likewise, if corrections such as those applied by this module were ever incorporated into the Perl core (so that Perl's built-in stat(), lstat() and utime() functions get/set correct UTC values themselves, even when built on the faulty Microsoft C library functions) then the corrections applied by this module would not be necessary.
In either case, this module would either need updating appropriately, or may even become redundant.
In fact, as noted above (see "A Happy Ending"), this module became mostly redundant with the release of Visual Studio 2015 (VC14.0), and wholly redundant with the release of Perl 5.33.5.
Patches, bug reports, suggestions or any other feedback is welcome.
Patches can be sent as GitHub pull requests at https://github.com/steve-m-hay/Win32-UTCFileTime/pulls.
Bug reports and suggestions can be made on the CPAN Request Tracker at https://rt.cpan.org/Public/Bug/Report.html?Queue=Win32-UTCFileTime.
Currently active requests on the CPAN Request Tracker can be viewed at https://rt.cpan.org/Public/Dist/Display.html?Status=Active;Queue=Win32-UTCFileTime.
Please test this distribution. See CPAN Testers Reports at https://www.cpantesters.org/ for details of how to get involved.
Previous test results on CPAN Testers can be viewed at https://www.cpantesters.org/distro/W/Win32-UTCFileTime.html.
Please rate this distribution on CPAN Ratings at https://cpanratings.perl.org/rate/?distribution=Win32-UTCFileTime.
"stat" in perlfunc, "lstat" in perlfunc, "utime" in perlfunc;
File::stat, Win32::FileTime.
Many thanks to Jonathan M Gilligan <jonathan.gilligan@vanderbilt.edu> and Tony M Hoyle <tmh@nodomain.org> who wrote much of the C code that this module is based on and granted permission to use it under the terms of the Perl Artistic License as well as the GNU GPL. Extra thanks to Jonathan for also granting permission to use his article describing the problem and his solution to it in the "BACKGROUND REFERENCE" section of this manpage.
Credit is also due to Slaven Rezic for finding Jonathan's work on the Code Project website (https://www.codeproject.com/) in response to my bug report (Perl RT#18513).
The custom import() method is based on that in the standard library module File::Glob (version 1.23), written by Nathan Torkington and others.
import()
The private _fixup_file() function is based on code taken from the win32_stat() function in Perl (version 5.19.10).
_fixup_file()
win32_stat()
The alt_stat() function is based on code taken from the wnt_stat() function in CVSNT (version 2.0.4) and the win32_stat() and pp_stat() functions in Perl (version 5.19.10).
wnt_stat()
pp_stat()
The Win32UTCFileTime_StrWinError() function used by the XS code is based on the win32_str_os_error() function in Perl (version 5.19.10).
Win32UTCFileTime_StrWinError()
win32_str_os_error()
The latest version of this module is available from CPAN (see "CPAN" in perlmodlib for details) at
https://metacpan.org/release/Win32-UTCFileTime or
https://www.cpan.org/authors/id/S/SH/SHAY/ or
https://www.cpan.org/modules/by-module/Win32/.
The latest source code is available from GitHub at https://github.com/steve-m-hay/Win32-UTCFileTime.
See the INSTALL file.
Steve Hay <shay@cpan.org>.
Copyright (C) 2003-2008, 2012-2015, 2020 Steve Hay. All rights reserved.
Portions Copyright (C) 2001 Jonathan M Gilligan. Used with permission.
Portions Copyright (C) 2001 Tony M Hoyle. Used with permission.
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself, i.e. under the terms of either the GNU General Public License or the Artistic License, as specified in the LICENCE file.
Version 1.60
23 Dec 2020
See the Changes file.
To install Win32::UTCFileTime, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Win32::UTCFileTime
CPAN shell
perl -MCPAN -e shell install Win32::UTCFileTime
For more information on module installation, please visit the detailed CPAN module installation guide.