Alternate Data Streams
Hidden data in NTFS
Both files and folders in NTFS file system can have additional “forks” of data called alternate data streams (ADS-es in short). If this reminds you of forks on Mac then this is no coincidence: this feature was added to Windows NT explicitly as a support for compatibility with Mac’s HFS. In general, the ability to add additional named attachments to any file of folder is a wonderful idea. Except that on Windows those attachments don’t show up in a standard folder listings (even on command line), and, worse than that, their contents do not contribute to reported file sizes. So, for instance, you can have an innocuous looking small text file with 1GB alternate data stream. Good luck hunting for your missing disk space. The fact that such alternate streams will be dropped when moved/copied out of NTFS is just a topping on a cake. Do not store anything important in an ADS. It is needless to say that ADS-es are also a huge security risk. You can even store hidden executables inside ADS-es.
On the other hand, they do have a lot of legitimate uses and they are actually used by some popular applications and even some Windows services.
E.g., SQL Server uses ADS in database file to store snapshot data,
Windows attachment manager stores web content zone information in an appropriate ADS, and you can make Dropbox ignore subfolder, or a file inside Dropbox folder, by placing 1 in the com.dropbox.ignored ADS of that file or folder. Thus, the ability to set, query and delete ADS-es may be of some practical value. It is also a great opportunity to learn something new about powershell.
Manipulating ADS-es in powershell
Let us create a new file (test.txt) in some location, and fill it with initial data using powershell command:
Set-Content -Path test.txt -Value 'Hello World'
We can verify that test.txt has size of 13 bytes. Let us now create an alternative data stream of test.txt called aedh, and fill it with some random data:
$yeats = @"
Had I the heavens' embroidered cloths,
Enwrought with golden and silver light,
The blue and the dim and the dark cloths
Of night and light and the half light,
I would spread the cloths under your feet:
But I, being poor, have only my dreams;
I have spread my dreams under your feet;
Tread softly because you tread on my dreams.
"@
Set-Content -Path test.txt -Value $yeats -Stream aedh
When we check the file’s metadata with
Get-Item -Path .\test.txt
we see that the file size did not change (although LastWriteTime was appropriately updated). When we display the contents of `test.txt’ with
Get-Content -Path .\test.txt
it also contains just Hello World. We would see the same if we opened it with notepad.
To see the aedh stream metadata or to view its contents we need to add -Stream option:
Get-Item -Path test.txt -Stream aedh
Get-Content -Path test.txt -Stream aedh
Some commands (including commandlets we used so far) may accept file names which include stream name after the colon, e.g.,
Get-Content -Path ./test.txt:aedh
Unfortunately, the support for such “stream aware file names” is not universal. Amazingly, notepad ./test.txt:aedh works, but
code ./test.txt:aedh does not. -Stream argument of Get-Item works with globs, e.g., use
Get-Item -Path test.txt -Stream *
to display all streams associated with test.txt. In this case you will see also :$DATA stream, i.e., the ordinary contents of the file. In order to see
the number of streams in the file and the total size of file including alternate streams one can use Measure-Object commandlet to count and sum lengths of streams received through pipe from Get-Item (note the backticks to mask new-lines):
Get-Item -Path ./test.txt -Stream * `
| Measure-Object -Property Length -Sum `
| select Count, Sum
Alternate data streams do not survive zipping: Compress test.txt, uncompress it, and then display data streams of uncompressed file:
Compress-Archive -Path test.txt -DestinationPath test.zip
Expand-Archive -Path .\test.zip
cd test
Get-Item -Path ./test.txt -Stream *
and you will notice that aedh is no longer there (it is, of course, still attached to the original file).
In order to actually remove ADS from file use Remove-Item with appropriate arguments, e.g.,
Remove-Item -Path test.txt -Stream aedh
Alternate data streams on folders
One can attach alternate data streams to folders as well as to files. One significant difference is that on folders ADS-es are not “alternate”,
but the only data streams, and this has consequences. If cat is a folder without any ADS-es attached, then Get-Item cat -Stream * displays nothing. This is one of the reasons why Get-Item * -Stream * is a poor replacement for
dir /R which works in cmd displaying in separate lines all alternative data streams for all folders and files in the current directory in addition to ordinary information for those files and folders.
Another bad thing about ADS-es on folders is that until powershell 7.2 Get-Item would not display information about them, -Stream option or not.
On the other hand, Set-Content and Get-Content worked fine with streams on folders.
Some applications
Zone.Identifier
Most web browsers under Windows (including Chrome, and, of course, Edge) will attach to every downloaded file an alternative data stream named Zone.Identifier. Long time ago Zone.Identifier would contain just a number
denoting web content zone. Web content zone 0 means
“downloaded from local computer”, 1 is “downloaded from intranet”, 2 “downloaded from trusted site”, 3 “downloaded from internet”, and, 4 “downloaded from somewhere Windows deems shady”. Nowdays, Zone.Identifier still contains web content zone, but also a lots of other things. For example, I have just downloaded (using Chrome)
some sample data from
one of pytorch tutorials. The command
Get-Content .\data.zip -Stream Zone.Identifier
returns
[ZoneTransfer]
ZoneId=3
ReferrerUrl=https://pytorch.org/
HostUrl=https://download.pytorch.org/tutorial/data.zip
Yes, this is INI file format. Above, HostUrl is obviously the site the file was directly downloaded from (with the full path). However, the link I clicked
was on page ReferrerUrl.
Needlesly to say, this is kind of creepy, though I do understand why all of this info might become useful when tracking provenance of malware. It is therefore somewhat reassuring that when I downloaded the same file from incognito tab, the content of Zone.Identifier was far less intrusive:
[ZoneTransfer]
ZoneId=3
HostUrl=about:internet
The standard Windows unzipper (called from explorer) attaches Zone.Identifier to each file from the uncompressed archive with contents like that (I have censored my username):
[ZoneTransfer]
ZoneId=3
ReferrerUrl=C:\Users\***Censored***\Downloads\data.zip
Thus, it still has the the ZoneId indicating this file comes from the internet.
However, the only url is a path to the archive. So this is less concerning, especially if you later delete original zip file. Interestingly, neither Expand-Archive nor tar does the same thing.
In order to run some downloaded exe file which Windows does not like, or simply to remove some privacy violating info, one can use Remove-Item, e.g.,
Remove-Item -Path data.zip -Stream Zone.Identifier
This is, however, such a common task that there is a specialized commandlet provided which does just that: Executing
Unblock-File -Path data.zip
removes Zone.Identifier. Since Unblock-File like Remove-Item accepts file names through the pipe (and by parameter name), we can get rid of all Zone.Identifier’s from all the files in the current folder as follows:
Get-ChildItem -Path . -Recurse -File | Unblock-File
Above, the -File flag causes Get-ChildItem to output only files.
Finally, if for some unfathomable reason you want to copy Zone.Identifier from one file to another you can do it as follows:
Get-Content -Path file.a -Stream Zone.Identifier `
| Set-Content -Path file.b -Stream Zone.Identifier
Dropbox
Say, you store on Dropbox your node project. And since disk space limit on free Dropbox account is rather small, you obviously do not want Dropbox to save in the cloud hundreds of megabytes of node_modules in addition to your actual sources. Problem is, node_modules must be inside your project, i.e., inside Dropbox. Can you make some files or folders deep inside Dropbox folder on your computer invisible to Dropbox? Yes you can. Dropbox will ignore any file or folder which has an alternate data stream named com.dropbox.ignored with 1 inside. Thus, to avoid syncing to cloud all the external modules downloaded through npm, do the following:
Set-Content -Path ./node_modules -Stream com.dropbox.ignored -Value 1
Actually, you can do it even less verbosely (it will also work in cmd):
echo 1 > ./node_modules:com.dropbox.ignored
Be warned that the above works in cmd and powershell 7.3. It did not work in powershell 5.1.