In the bad old Linux days, if you read a huge amount of data off the disk (like if you were doing a backup) Linux would try to take all that data and jam it in the page cache. This would push out all the useful stuff in your cache, and sometimes even cause swapping as Linux helpfully swaps out stuff to make room for more useless page cache.
One of the great things about `dd` is that you have a lot of control how the input and output files are opened. You can bypass the page cache when reading data by using iflag=direct, which would stop this from happening.
Moreover, flash drives (and all flash media) have a favorite page size, which is generally 4kB or 512kB. By defining a common denominator page size like 1024kB (with bs=1024kB), you can keep your flash drive happy with enough backlog to write, so it can perform at its peak write speed without churning, which will help with faster writes and lower write amplification, which is a win-win.
Man, this brings back memories of a few years ago when I tried to dd a Linux install image on a usb drive.
It would take forever and always end up with an i/o error. I figured the new PC had somehow wonky usb ports or something. But when that happened on another "known good" box, I figured it was the flash drive. But Disk Utility in MacOS worked well.
Then I tried increasing the block size to 1M and everything went smoothly. It even took less time to write the whole image correctly than it took it to error out before.
(In theory at least,) kernel should take care of aggregating write blocks and those will be plenty big enough by the time it reaches target drive, all thanks to the very same page cache GP is talking about - unless you specify "oflag=direct" to dd.
That being said, probably don't use too small of a block-size - this will eat up CPU in system call overhead and slow down copy regardless of target media type.
I don't know how internals of "cp" and the related machinery interacts with the target drive, however if you don't provide bs=1024kB, dd writes with extremely small units (1 byte at a time IIRC), which overwhelms the flash controller and creates a high CPU load at the same time.
I always used dd since it provides more direct control over the transfer stream and how it's transported. I also call dd as "direct-drive" sometimes due to these capabilities of the tool.
"By specification, its default 512 block size has had to remain unchanged for decades. Today, this tiny size makes it CPU bound by default. A script that doesn’t specify a block size is very inefficient, and any script that picks the current optimal value may slowly become obsolete — or start obsolete if it’s copied from "
While I remembered the default wrong (because I never used the defaults, and I was too lazy to look for it during writing the comment), it's possible for a script to get a correct block size every time.
There are ways to get block size of a device. Multiply it by 2 to 4 (or more), open it directly, and keep your device busy.
The blog post is oblivious to nuances about the issue and usefulness of "dd" in general.
Please forgive the nit-picking, I'm not attacking this (excellent) article, or your entirely sensible inclination to dig up some "physical" number, but...
With modern SSDs, "sector/block size" is rapidly approaching vagueness of cylinder/head/sector addressing scheme, as used a couple of decades ago on venerable spinning/magnetic disks.
That is - it is definitely a thing, somewhere deep down, but software running on host CPU trying to address those, wouldn't necessarily end up addressing the same thing as user had in mind.
If you want a concrete example - look no further than "SLC mode" cache - where drive would have a number of identical flash chips, but some of them (or even a dynamically-allocated fractional number of chips) would be run at lower bits per cell count, for higher speed/endurance. However erase- and write- blocksize for the chip is expressed in cells, not bits. What that means is - cache and main storage of the very same SSD would have different blocksize (in bits/bytes).
> Please forgive the nit-picking, I'm not attacking this (excellent) article, or your entirely sensible inclination to dig up some "physical" number, but...
I don't think it's nitpicking. We're discussing there. We're technical people, and we tend to point out different aspects/perspectives of a problem, and offer our opinions. That's something I love when it's done in a civilized manner.
Regarding to remaining part of your comment (I didn't want to quote to not make it look crowded), I kindly disagree.
The beauty of SSDs are they have a controller which fits into the definition of black magic, and all flash is abstracted behind that, but not completely. Hard drives also are almost in the same realm.
Running a simple "smartctl -a /dev/sdX" returns a line like the following:
This means I can bang it with 512 byte packs and it'll handle it fine, but the physical cell (or sector) size is different, which is 4kB. I have another SSD, again from same manufacturer, which reports:
Sector Size: 512 bytes logical/physical
So, I can dd it and it'll just handle it just fine, but the first one needs a bs=4kB to minimize write-amplification and maximize speed.
This is completely same with USB flash drives. Higher end drives will provide full SMART (since they're bona-fide SSDs), but lower end ones are not that talkative. Nevertheless, a common denominator block size (like 1024kB, because drives also can be composed of huge, 512kB cells, too) allows any drive to divide the chunk to physical flash sector sizes optimally, and push data to it
In the SLC/xLC hybrid drives' case, controller does the math to minimize the write amplification, but again having a perfect multiple of reported physical sector size makes controller's work much easier, and makes things way smoother. Either because the reported physical size is for the SLC part which is you're hitting for the most cases, or the controller is already handling multi-level logistics inside the flash array (but thinking in terms of block sizes since the this is how it works on the bus side regardless of the case inside).
It's great to have control over this but I suspect most users never knew this was happening, had no idea dd could bypass that behavior, nor knew which argument to pass to dd to accomplish this. It's like saying 'what makes 3d printers so great is you can make anything!' but you'd be way better off with an industrially forged object than the 3d printed object.
That's cool, I didn't know you could do that. How do you make the mold? If (as I suspect) the answer is handcrafting, 3d printing still offers a considerable advantage in labor.
Oh gosh yes. Though for certain finishes, vacuum forming still has some upsides. Whats neat is you can use 3D printing to make the mould for vacuum forming over.
This gives you the best of both worlds, in terms of the shapes/finishes that forming can achieve, but with the workflow benefits of 3D printing.
Similarly, the new hotness is compression/cold forging using 3D printed moulds and carbon fibre (and other materials). Very very cool.
Pretty much, and understanding what is going on "under the hood" as it were can be informative. Had the author done a 'cp myfile.foo /dev/sdb' on a UNIX system they would have found they now had a regular file named '/dev/sdb' with the contents of myfile.foo and and their sd card would have remained untouched. But you would only know that if you realized that cp would check to see if the file existed in the destination, unlink it[1], and then create a new file to copy into.
The subtlety of opening the destination file first, and then writing into it, was what made dd 'special' (and it would open things in RAW mode so there wasn't any translation going on for say, terminals) but that is lost on people. Bypassing the page cache and thus not killing directory and file operations for other users of the system is a level even below that. Only the few remaining who have done things "poorly" an incurred the wrath of the other users of the system sitting in the same room really get a good feel for that :-). Fortunately for nearly everybody these days they will never have to experience that social embarrassment. :-)
[1] Well unless you had noclobber set in which case it would error out.
Still the bad old days for copying files from iOS to Linux. It seems to make a copy internally of everything you copy internally on the device before sending it, which leads to running out of free space just trying to copy things of :(
GP is talking about the Linux kernel's buffer cache. Unless you tell the kernel to operate directly on the disk, your reads come from and your writes go to pages within the kernel's buffer cache. Using a small bs probably results in a buffer of only bs bytes inside dd's address space, but the buffer cache is completely different and resides in the kernel.
That is, without iflag=direct, dd will repeatedly ask the kernel to copy bs bytes from the kernel's buffer cache into dd's address space, and then ask the kernel to copy bs bytes from its address space into the kernel's buffer cache.
That should write 512 bytes from "source" beginning 10k into "target". (I've not tested this, you should verify.)
4. Create a sparse file. Sparse files appear to have a nonzero size, but take up no space on disk, until data is actually written to them. These are often used as "inflating" dynamic filesystem images for virtual machines.
When reading to or from IBM data tapes, you might find blocking / unblocking conversions useful. I've done this, but it's so long ago that I don't trust my memory on that any more. Odds are good you'll not have to worry about this.
There are other useful applications as well, though these are not typically encountered very often. Do feel free to explore and attempt these on safe media.
Careful, your #2 and #3 are incorrect -- skip and seek operate with blocks, not bytes. So your #2 would copy 32 bytes after the first 32kB of data, and #3 would write 512 bytes at position 5120k.
Especially handy when you've fed in a huge amount of JSON (sometimes all on one line because, y'know, why not) into jq and you get the inscrutable output:
parse error: Invalid numeric literal at line 1, column 236162512
A note here: if you're using GNU dd, you can also use iflag=count_bytes and then set the block-size to whatever you want. That'll give you the best of both worlds.
The “section”[0] or “subpage.” Programs are section 1, hence ‘dd(1)’ The idea is that it can be possible for a library function (section 3) to have the same name as an executable. In which case, you’d type into your shell:
man 1 <name> # executable: <name>(1)
man 3 <name> # function: <name>(3)
Without the distinction argument, man will throw its arms up in the air and give up.
However, if there’s no ambiguity (such as with ‘dd’ where only the executable exists), you can drop the section number parameter when running man. man will then search all the sections, see that ‘dd’ only exists in section 1, and go from there.
Something like `read` gives a better example. Like `man 1 read` is "read — read from standard input into shell variables", "read [-r] var...", a shell function. And `man 2 read` is " read - read from a file descriptor", "ssize_t read(int fd, void buf, size_t count);". You can also get into `man 8 catman`, "catman - create or update the pre-formatted manual pages", "catman [-d?V] [-M path] [-C file] [section] ...". The `catman` is a mirror-like hierarchy of man pages (usually in troff format using the 'an' macro package) pre-formatted for the standard terminal or whatever to shave off a bit of time running roff on the source files all the time.
This is all ancient knowledge and well in place back in the mid 1980's long before Linux or any of that stuff. The `man` page sections used to be different 3-ring binders all printed out sitting on a table in the computer lab. The 'sections' were just different binders of documentation. Get off my lawn!!!
I also use the notation as a convention to indicate that I'm referring to a Unix command (or library function, etc.). E.g., "cat" might refer to a feline, but "cat(1)" should more clearly refer to the Unix / Linux command.
(Where I've got proper markup, I'll typically set commands or references in monospace using backtick notation: `dd`, `cat`, etc.
Yep. Executables are section 1 simply because they came first in the binders.
There's nothing necessarily preventing man from allowing string codes as a replacement, but man interprets (fully) non-numeric arguments as pages to go. So `man cat dd` (on Ubuntu) will first open cat(1), and when you exit (with 'q'), will prompt if you want to continue. If you say yes, it'll open dd(1).
That has the side effect than `man exe dd` would be interpreted as opening exe(#) (printing "No manual entry for exe") followed by dd(1).
Man pages are divided into sections as follows (GNU & Linux):
0 Header files (usually found in /usr/include)
1 Executable programs or shell commands
2 System calls (functions provided by the kernel)
3 Library calls (functions within program libraries)
4 Special files (usually found in /dev)
5 File formats and conventions, e.g. /etc/passwd
6 Games
7 Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)
8 System administration commands (usually only for root)
9 Kernel routines [Non standard]
GP’s comment should apply to most if not all systems with a few caveats / divergences.
FreeBSD certainly mentions and lists standard sections at the top of `man man`. OpenBSD mentions the concept of categories / sections early on, though it only lists the specific section when it comes around to documenting the corresponding filter.
It is just a pet peeve of mine that a lot of younger pros and devs talk about Linux as though it were groundbreaking, when all of GNU/Linux is a copy of a copy of a copy of a copy of the actual earth-shaking developments, SysV (arbitrary) and BSD. Everything in Linux was there before Linux was a twinkle in Linus' eye. I remember when most webservers ran NetBSD, and 5 years later Linux took over the data center. Was NetBSD really so intolerable and Linux really that superior? No, its just that with no memory of the past, one can't know any better. Linux really brought nothing new, no new advances, nothing that wasn't there before, and yet it took over like a jihad. That isn't accurate... not like a jihad... it was a jihad. We can thank fanaticism for Linux in the datacenter. I'm not unfaithful, just system-agnostic. Linux was a solution to a problem already solved many times. And since Linux is not original, it sort of gets under my skin when it is insinuated as such. When talking about any software, one should refer to its original development. When we talk about web browsers, we don't talk as though, say, Microsoft invented the web browser, just because it has one; instead we talk about Tim Berners-Lee, not a copy of a copy of a copy of his work.
The main reason Linux outcompeted the BSD-descendants is the GPL license. Instead of competing with proprietary extensions on a free base, fragmenting the ecosystem, upstreaming as much work as possible makes economic sense.
We can still observe that effect in the Android ecosystem which started out really bad and fragmented and slowly drifts into a more coherent whole, instead of the other way around.
Many saw what the UNIX war led to and tended to avoid similar situations. I too liked what the BSD homogenous distribution let to on a technical basis, but the GPL makes sense as long as business cases can be made fit.
Nah the main reason Linux outcompeted the BSD is that it arrived at exactly the worst moment for the BSDs: USL v. BSDi.
This case put a severe pall on the attractiveness of BSDs as they were suddenly in legal jeopardy just at the outset of the UNIX wars and as they were coming into their own.
And at the same moment, a cleanroom unix arrived on the market, limited in many ways but safe.
The GPL was at best neutral for most users, as can be seen from its adoption (or lack thereof). However the GPL was nowhere near as problematic as “AT&T might get our OS declared illegal”.
Yes, there was that, too. Now I did not mean that the license is of crucial importance to most end users (it might be for some, but likely the other way around as some will prefer the simpler BSD clauses). But it was decidedly important for the business and consulting side to form, and that was hugely where Linux won.
Red Hat, Cygnus and the IBM service group took early big bets on Linux, which could not have happened on a product where vendors based their respective offerings on proprietary lock-ins. That drove adoption in banking and defense whose existing UNIX stacks looked increasingly old, which drove a huge industry shift that took the better part of a decade.
It used to be quite common to find people arguing that BSD was the more "business friendly" license, which is may be true in some specific ways but tends to miss the bigger picture. The adoption of a mainstream system under GPL license was important.
Then the situation was probably different in the web hosting business, in academia, and in other sectors where other factors dominate.
What NetBSD did not have was drivers for any old commodity PC hardware which everyone had laying around. That’s it. That’s why people ran Linux on their stuff, and then continued running Linux in the data center.
386BSD the predecessor of Net/Free/OpenBSD was published under a BSD license in 1992. 6 months after Linus posted his kernel sources on usenet.
It was free and open source and no later than the "free" BSDs that are still available today. I'd say that is a reason why it is successful.
Edit: "free" in quotatio marks to not confuse with FreeBSD.
I fail to see how the comment you are so vehemently responding to in anyway implied that GNU/Linux was somehow groundbreaking. The only mention of GNU & Linux was to clarify that the sections that followed are the man sections on GNU/Linux systems which is appropriate since not all UNIX flavors contain the same sections or the same order of sections. GNU/Linux systems mostly utilize the same sections and order as BSD based systems. SYSV based systems usually have some differing sections and a differing order of sections.
> We can thank fanaticism for Linux in the datacenter. I'm not unfaithful, just system-agnostic. Linux was a solution to a problem already solved many times
And outside the datacenter? Was the problem of an Open Common Desktop OS - for those people who are radically *not* system-agnostic - solved at the time?
What could have been the effects on the trends building the scenario until today and beyond, had Linux not appeared but keeping the rest of the chessboard intact?
On the topic of hidden gems. The info directory is frequently populated by default with manuals when you install a piece of software. These tend to be more in-depth and complete manuals then the man pages for the same tool. Just type `info` in a terminal and be amazed. Emacs has a convenient info viewer also, under `C-h i`.
Perhaps more directly to your question: Yes, as siblings note it is a manpage section number, but writing it like that is just a way to refer to programs; "cat" could be a feline, but "cat(1)" is a unix program. Oh, and it can disambiguate; printf(1) is a program you run from the shell, printf(3) is a C library function. IMO it's as much a cultural convention as anything.
the other answers are good, but haven't quite covered the topic.
the unix manual was commonly printed out, and these section numbers were a necessity for looking things up. Collation was section number, and then alphabetical
when you were first introduced to unix, you sat down with the manual and read it.
What drives whoever's the second person to implement such a flag to make it different? An explicit desire to inconvenience other "tribes" of computer users and use the flag as a symbol of an in-group?
Apparently a typo on the post I was replying to...there is no "-b" switch, it's "-c" or "--bytes" for gnu head. Though there are versions of head without one or both of "-c", "--bytes".
caveat operator: to ensure the conv=sparse operand achieves the desired outcome, be sure to use an output blocksize equal to st_blksize of the output filesystem.
Yup, `dd` helps in a lot of situations when you need portability, not just this one.
Commands like `cp --sparse=always` aren't worth such a blanket disrecommendation though; if you are working directly at a console as opposed to scripting you typically don't need portability.
My most common use for `dd` is using it with `sudo` to direct the output of a unprivileged pipeline to a root-owned file. Instead of running `echo hello >/root/test.txt`, which will fail, I use `echo hello | sudo dd of=/root/test.txt`.
a note: I'd recommend using tee instead of dd for that job, or add iflag=fullblock if your dd supports it.
The thing is that dd issues a read() for each block, but is doesn't actually care how many bytes it gets back in response (unless you turn on fullblock mode).
This isn't really a problem when you're reading from a block device, because it's pretty uncommon to get back less data than you requested. But when you're reading from a pipe, it can/does happen sometimes. So you might ask for five 32-byte chunks, and get [32, 32, 30, 32, 32]-sized chunks instead. This has the effect of messing up the contents of file you're writing, with possibly destructive effects.
To avoid it, use `tee` or something else. Or use iflag=fullblock to ensure that you get every byte you request (up to EOF or count==N).
I suspect you meant -c 440; I can’t find a variant of head(1) that has a -b operand on any Unix. Note that -c is not POSIX but does have widespread support. Notably missing on Solaris.
Fun fact, the -c usage comes from ksh, where head is a shell builtin.
The trick to getting this throughput is telling Linux to do an insanely large IO (4 MiB). The drive can't do 4 MiB reads - the largest IO it can handle is 2 MiB.
My SSD - apparently one of the fastest on the market - KINGSTON SKC3000D2048G - rated 7GB/7GB read/write, after writting data to it, stopped being very fast. No idea, if it's something related to how nvme/ssd-s work, or maybe I have a broken unit.
I have another nvme drive - Force MP510 - and it doesn't care if data was previously written or not. When reading from it, I get ~full x4/Gen3 speeds of 3.5GB/s
PS: nvme smart-log shows 100% available spare, and 0% percentage_used, so it doesn't seem to be wear-related.
They write in small blocks (eg, 4K), but can only erase in very large blocks (eg, 2M). Writing to an empty device is easy, but eventually you have to overwrite something, and you can't just replace a 4K block with another. You have to take a contiguous block of 2M, wipe the entire thing, rewrite whatever part of it was useful, then do the write you actually wanted to.
This effect is known as "write amplification", and it means that in bad cases you need to do many times more work than the host system requested.
Modern high end SSDs have various ways of dealing with this like a RAM cache, a SLC cache and extra reserved space to always have some spare room, but there are still limits.
This is what TRIM (confusingly also known as 'discard' in some contexts) is for -- the SSD operates in blocks and has no way to know that some chunk you've written to before is now useless to you because you deleted the file it belonged to, until you give it the command to overwrite that block. TRIM tells the drive "these parts can be recycled", and allows it to create empty blocks in advance. So make sure to TRIM once in a while.
Sadly TRIM wasn't well specified initially in regards to what performance characteristics it should have -- some old drives can get stuck on it for a while. So while many filesystems support emitting TRIM automatically, it can cause severe performance issues on some drives and the recommendation is to do it as a maintenance task on a timer instead.
TL;DR: Run `fstrim /mountpoint`. Wait a while before testing if it changed anything. The drive isn't obligated to do the work immediately.
It gets a bit more complex than that due to layers. LVM and dmcrypt can filter out TRIM requests. You can use `lsblk -D` to check the support status.
> You have to take a contiguous block of 2M, wipe the entire thing, rewrite whatever part of it was useful,
This is called garbage collection. It may be happening at any time in the background but becomes more frequent and perhaps in the write path as the drive fills. 2M is an example size - the actual size will vary by drive model and it is rarely disclosed.
> Modern high end SSDs have various ways of dealing with this like a RAM cache, a SLC cache and extra reserved space to always have some spare room, but there are still limits.
There are multiple types of SLC cache as well. Client drives may have a small number of gigabytes of SLC that can absorb a small burst of writes. Client drives may also have pseudo SLC (pSLC) that is called pSLC, TurboWrite (Samsung), etc. With pSLC, when there's about 30% of the drive's NAND that is erased, the drive will use that as SLC. So, a 1 TB drive will use about 300 GB of space as a 100 GB SLC cache.
How performance degrades as the drive fills varies widely between drive models. Some drives start to have significant read and write performance degradation long before the drive is half written. Others will maintain fairly consistent read performance (maybe within 90% of that which is advertised) regardless of how full the drive is. For instance, the original version of the Samsung 980 Pro maintains close-to-spec read speeds regardless of how full it is but write performance drops from about 5200 MB/s to about 1300(?) MB/s the moment it hits 70% allocated.
Datacenter and enterprise class drives tend to have lower peak performance than client drives, but their performance is much more consistent regardless of how full they are.
If you are buying a client NVMe drive for speed, buy one that is larger than you need and set aside at least 30% of it in unpartitioned (or unused partition) space. This will prevent the OS from writing to 30% of the drive thus keeping plenty of space for pSLC and similar optimizations. This will also increase the life of the drive as garbage collection is likely to have to rewrite the same data less frequently, resulting in a lower write amplification factor.[1]
I've found that the write pattern can have an effect on future reads. For instance, I've seen random 4k writes followed by 1m reads result in a somewhat lower read rate than if the writes were originally done as 1m writes.
The degradation you are seeing is rather extreme - it seems to be performing worse than a cheap SATA SSD. I'd say this is worth having a conversation with Kingston.
Oh thank you for submitting this to HN. I’ve been telling people not to use dd for years and everyone looks at me like I just gave birth to a full grown dinosaur or something.
“Well why does the entire internet say to use dd then?”
Because they copy from each other just like you copied from them. Just use cat.
More likely is you are messing around with some new/infrequently used utility and you pass in the arguments incorrectly and specify "file" as the output instead of the input as intended.
The one I typed is unlikely (I wrote it in response to your "Anything after the `cat file |` can't overwrite the file", which is factually incorrect as demonstrated), but there are plenty of other simple mistakes. Understand what you're running, and keep backups, otherwise you'll screw up eventually. Selecting your tools this way doesn't move the needle much.
Well, yes, but then it wouldn't have been a very good demonstration of why the statement "anything after the `cat file |` can't overwrite the file" is incorrect.
So Grandma's Ham is a variant of Chesterton's Fence... When you do or don't do something for reasons of tradition, find the real reasons for doing / not doing it.
It's not often I actually unzip stuff on the command line these days, but saying "extract ze files" in a terrible French accent in my head when I do is a highlight.
Yup, I remember when you had to pass `z` if it was gzip, and I remember my surprise when I missed it once and it still worked (apparently about 5 years after it was no longer needed!)
"Well why does the entire internet say to use dd then?"
Because you usually need root to write to drives, and you can't `sudo` redirect. `sudo dd if=whatever.img of=/dev/whatever` is nicer and easier than `sudo sh -c "cat whatever.img >/dev/whatever"`.
In the cases where you need root it's often even easier to use the 'tee' command instead of spawning another shell; esp. if the command has symbols which need to be escaped.
Just like in household plumbing, the 'tee' command basically takes the input and sends it to more than one place. Naturally, running 'sudo tee' will let you send things all over but as another user.
All that said, I won't speak to the comment "why does the entire internet say to use dd". I've never actually found the "whole internet" to agree on much of anything (-:
Well, I’d like to thank the author for writing this. I always enjoy having the veil of “magic” lifted from computers.
UNIX’s simplicity of “everything is a file” continues to surprise me, even though it shouldn’t any more. I think a lack of confidence in my understanding of the basics, leaves me tempted to copy from others as you mention.
What if you need to test direct IO, or test various partial-page, partial-block, or full block/page access to pagecache? Sometimes you might actually want to try using direct IO too, if buffered IO is not giving you sufficient control or reporting of errors from the device, or continuation from part-way through the copy, and in some cases possibly because you don't want to consume memory for page cache (although dd isn't good with direct IO so that's probably unlikely).
That said if you don't know whether or not you need it, you don't. And those who do would know not to listen to your advice (for what they need to do) anyway, so it's not bad general advice to give.
Direct I/O is notoriously finicky. Depending on what filesystem you're using there are apparently alignment restrictions and other corner cases to think about. I don't think it's that `dd` isn't good with Direct I/O, I just think its hard to use.
Direct IO is not finicky (on Linux, since 2.4 era kernels). It's just that at least with dd, if you don't know what it does then it probably does not do what you want (synchronous, single queue depth). If you know what you want and know what it does, then it's fine to use, and can be useful (I used it today to measure block device interrupt latency in a VM).
> Usage of dd in this context is so pervasive that it’s being hailed as the magic gatekeeper of raw devices.
That's the thing; it isn't. Author forgot to explain (if he knows that at all) that /dev/sda2 on Linux is not a raw device. It's a block device.
So if dd is hailed as something to use on /dev/sda, that's not an example of being hailed for a raw device.
dd's capability to control the read/write size is needed for classic raw devices on Unix, which require transfers to follow certain sizes.
E.g if a classic Unix tape needs 512 byte blocks, but you do 1024 byte writes, you lose half the data; each write creates a block.
The raw/block terminology comes from Unix. You have a raw device and a block device representing the same device. The block device allows arbitrarily sized reads and writes, doing the re-blocking underneath. That overhead costs something, which you can avoid by using the raw device (and doing so correctly).
In a way, but in Linux block devices can still be accessed as block devices, while in FreeBSD (since around 1999) they can't - there's no caching at that level anymore; raw devices (which Linux got a few years before that) are the only kind of devices. (If you do "ls -al /dev", you'll still see block devices, but it's maintained only to pretend to userspace, just like major/minor numbers.)
Tapes aren't block devices at all - that's why you can't mount them :-)
Mounting and a device being a block device or not had zero relation in classic unix though :)
Tapes by nature are block devices due to only being accessible in block increments, and writing them in transparent way is a bit more problematic than with disks, as being able to just re-read a block to do read-update-write cycle isn't guaranteed - or necessarily easy.
And yes, there used to be a time where you could mount tape as filesystem ;)
Heh, I remember finding out this on the good old days when I was finding out how to rip cds and shitting bricks. I mean I read it on a BBS and thought "that can't be right"; I was expecting to find something like the Nero suite back in windows.
Much more recently, I enjoyed the same kind of amazement on the bash tcp pseudo devices.
I think this has a bit of bad advice for using cp or shell redirection to read / write to raw block devices but dd isn't necessarily the best either. Personally any time I'm trying to image some hard drive, damaged or otherwise, I'll just about always jump straight to ddrescue (not dd_rescue). It's similar to dd, surprise surprise, but it keeps a log of which parts of the input and output have been copied / had errors / skipped. Nothing is more annoying than waiting an hour for some large copy to make progress and then run into an error or get interrupted for whatever reason. Using ddrescue because it keeps a log of the status of the operation you can resume it with the same command and it will pick back up right where it left of instead of having to start all over. It's also intelligent enough to not fail the first time and skip over some bad region of the disk on error and come back and reattempt it using various strategies once it's already copied the low hanging fruit.
There's very little reason not to use it, even if it's just to get a nice progress view instead of just the current amount of data copied.
Just a note -- despite the title, the article eventually presents a nuanced view and points out that "dd" has some uses.
> If an alias specifies -a, cp might try to create a new block device rather than a copy of the file data. If using gzip without redirection, it may try to be helpful and skip the file for not being regular. Neither of them will write out a reassuring status during or after a copy.
> dd, meanwhile, has one job*: copy data from one place to another. It doesn’t care about files, safeguards or user convenience. It will not try to second guess your intent, based on trailing slashes or types of files.
> However, when this is no longer a convenience, like when combining it with other tools that already read and write files, one should not feel guilty for leaving dd out entirely.
One thing dd does for me, that cp and cat do not, is that it forces me to read and check the command at least three times before pressing enter, which is a very good thing when messing with raw devices.
When I first learned that dd was not magical, I started using cp but I made some mistakes with partitions number and whatnot (nothing serious).
Maybe it's just the wierd syntax or the fact that I treat dd differently, I'm just more cautious and don't press enter automatically. Of course it's silly but to me at least, that's a good reason to keep using dd.
Back in the Dark Ages (1968) we had pre-written JCl scripts. I don't think many people knew JCL. We just appended a script to the front of our Fortran card decks. A DD card was the final card. After the frustrating work of finally getting the JCL right, the DD card was always just the "Do it Damn it" card in my mind.
This actually serves at least two approximately legitimate purposes: firstly, it ensures that reads from sda are aligned to a (at least nominal, ie 512-byte) disk block, which doesnt matter for normal, kernel-supported drives like IDE/SATA/most USB (which is almost certainly what sda is), but avoids bespoke devices (or their drivers) trying to do something clever when gzip asks for 1 or 7 or 17 bytes. (And writes to poorly-designed devices/drivers can be even worse.)
More importantly, like useless use of cat, it prevents gzip from trying to delete sda when it's done, which is something it will in fact do:
$ echo test > /tmp/sda
$ gzip /tmp/sda
$ cat /tmp/sda
cat: /tmp/sda: No such file or directory
(For gzip specificially, you can also prevent this by writing `gzip </tmp/sda`, but I've occasionally run into tools that try to 'intellegently' handle stdin file descriptors that point at 'real' files, so I feel better having a separate process blocking the way.)
gzip doesn't do a stat(2) and detect that /dev/sda is a device node? looks at gzip source code -- sure looks like it will delete device nodes. And seems like the wrong thing for it to be doing.
The underlying question, when dealing with stupid DWIM logic, isn't "Does it do that?"; it's "Can I be sure that it does that, and will continue to do that, even on old versions on legacy systems that a unknown amount of random crap depends on to not fall over because I accidentally the hard drive?".
And there's also the fact that deleting the source file by default is always the wrong thing for it to be doing. If I want to deal with corner cases like being almost out of disk space, I can pass --delete-source explicitly.
It’s funny (given that the name likely is inspired by Useless Use of ‘cat’) that examples here have actual useless uses of ‘cat’. ‘cat file | pv > disk’ can just be ‘pv < file > disk’.
But whatever. I’m sure 90% of code stems from something someone read and copied or came most easily to them. It works.
‘dd’ was really useful for finicky media like tapes and doing EBCDIC translation. It’s still great when you combine bs and count. Blow away an MBR, make an xGB file, etc.
It’s a Swiss Army knife. It can do a lot, but isn’t the best tool for most things. I still love it. Probably just muscle memory.
Original author here. The intended point was that `dd if=.. | something | dd of=..` is just as useless as the two cats in `cat .. | something | cat > ..`. People sometimes mock the latter while still believing that the former is necessary.
Sorry, that came off snarkier than I meant rereading it.
You’re 100% correct. I’ve grown more tolerant of these ceremonial uses. There really aren’t many folks that understand shell. I see it in CICD, etc all the time, too.
The one left that kills me that I see in vendor scripts all the time is ‘command; ret=$?; if [ “$ret” -ne 0 ]; then foo; fi’
If you’re handling return code cases, then cool. But folks don’t realize ‘test’ aka ‘[‘ is just another command.
But then again, so much Java, etc. I read is the same copy/paste. Those can be worse because example code isn’t prod ready, where bad shell usually does the right thing, just awkwardly or inefficiently.
But as you point out, so much isn’t thinking about what you’re doing, just mimicking.
So, in FreeBSD "dd if=/dev/ada0p1 of=/dev/null bs=1 count=1" will fail: disk driver will return EINVAL from read(2) because I/O size (1) is not divisible by physical sector size (usually 512). "cat" with buffer size X (which depends on the implementation) will either work or not depending on divisibility of X by physical sector size, and other random factors, like short file I/O caused by delivery of a signal.
Summary: dd(1) still has its place and author of original article is getting it wrong.
My best useless use of dd was in the Ubuntu 16.04 days.
I was at a satellite office with all windows PCs for the day, so used a live disk to get a decent environment to get things done.
Only problem was that the DVD drive kept spinning down and every time I did something that was not cached it made me wait forever.
nohup + while loop + sleep 4s + raw dd read from CD for the win :)
EDIT: Reading this article it sounds like dd has no "special" ability to access the disk in a raw way. But surely that's what the nocache option is for...
I’ve only used dd once in my life (not much of a hacker!) but it was mostly useless.
Dropbox had some promotion for their new photo storage service, where they were giving away free extra space up to 10gb to encourage you to store your photos.
Some clever cookie told me you could make 10gb of ‘empty’ jpeg files and put them in your Dropbox folder.
The Dropbox app would compress this 10gb of files for upload (usual behaviour for the app in those days) and increase your storage permanently for free.
I can’t remember the command but it was something like
Occasionally I run a dd on a loop with increasing block sizes to see what is actually fastest. I regularly see instructions on the web saying you should use "1M" or even "4M", but in my tests, smaller block sizes are often faster.
A few years ago, "128K" was usually the fastest choice. Today, on faster systems, "512K" has a slight edge.
I could not tell you why, though. Try it for yourself.
If you wirte to USB sticks / SD cards you should use (a multiple of) of the erase block size in oder not wear out the flash. I don't know what somewhat current sizes are. But 1M sounds likely to be a reasonable multiple.
For USB also make sure to power it off before removing otherwise you might lose data. Eg. by
Sync just syncs but it doesn't prevent anything from starting to write while you go to pull it.
"udisksctl power-off" checks nothing is using the drive, commits buffers to storage, deconfigures the drive, and powers it off. Unfortunately it may kill more devices than you want in some scenarios due to the way killing the port works. Also for normal USB drives I don't think® poweroff is any different than an immediate physical pull post unmounting. The upside is it'll be completely disabled the instance the command completes, i.e. it can't even be written to raw as it is no longer powered even though plugged in. Not sure how helpful that is in reality though.
I usually just umount the drive which syncs and prevents further writes (well, to the mount at least) but doesn't try to kill it for me. Most likely sync is "good enough" for the vast majority of use though.
Eject is supposed to be like the udisksctl option but I've had it act wonky on me depending on the type of drive being ejected so I've avoided it myself. If it works though it shouldn't be any different.
I feel more comfortable with just unmounting and powering off nowadays. But I have no hard evidence.
Hardly ever user more than one USB drive same time, so not sure whether there could be side effects with powering off one of them. Well, USB being USB, nothing would surprise me too much.
I understand his point, but using 'dd' allows you to set a buffer size which can make cloning a bunch faster. It also has great progress reporting (status=progress) which is really useful for the things dd is usually used for.
And even if you use it without it being needed, it's not a big deal. It doesn't add much overhead, if any.
By the way, on versions that don't support status=progress (like busybox and the BSD versions), you can periodically send a USR1 signal to get a progress update.
On the left side of a typical Linux hobbyist experience graph you probably have "this is a disk image file, and this is a disk, I'll just copy the disk image file onto the disk!", then you have a period of "I'll use the cool dd tool (without oflag=direct and/or sync because by this point you know people use it but you don't know why people use it) like I saw on the internet!", then when you understand that everything is a file you have "this is a disk image file, and this is a disk, I'll just copy the disk image file onto the disk!" again.
I personally suggest recommending people the Disks application included with Ubuntu Desktop and Fedora/Centos Workstation. It shows icons representing internal disks, SD cards or flash drives so they know what device they want to work with. If they want to take their time they can see all the information about the drives and partitions, they can start discovering and asking questions and reading up on how computers use disks right from there if they want to. And if they don't want to, it's just extra confirmations that it's the correct disk or DVD that they want to put their image onto. Then when they're sure about the device they can create a disk image and restore a disk image in that same application!
dd lets you specify the write block size. This is essential when writing to 9 track tape. Everything is a byte stream… except for the things which are not.
I don’t think the author is intending to tell everyone to always use something else, I think the author is trying to tell most people who use dd that there are easier ways to do what they are trying to do, and that things are files, which is something that a lot of people here seem to be forgetting.
"The O_DIRECT flag on its own makes an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and necessary metadata are transferred."
While I don't think this is relevant to block devices, I see no harm in including the flag in my alias, either.
I can't find anything about it in the man page - does anyone know the options? I'm assuming it doesn't mean simply running dd with seek to output another file and then running that.
Original author here. An example would be `{ dd bs=1 count=0 skip=1234; myprogram; } < file`. This is similar to C `lseek(0, 1234, SEEK_CUR); execve("myprogram", ...);`
It's not something you'd realistically use, but it's better to be limited by imagination than tooling.
Ah, I get it! What I initially thought you meant was creating a process from an ELF embedded further into the file. For example lseek-ing into a tar file and somehow execve-ing an included binary. Not I wonder if Linux you allow me to run /dev/stdin...
I used to used dd to convert EBCDIC to ASCII reading tapes from a 1/2" reel-to-reel tape drive. The "convert" capability of dd is what differentiates it from utilities such as cat.
dd is one of THE most common ways people accidentally wipe disks. All it takes is typing sda when you meant sdb.
The by-label trick helps a bit, but I still don't like it. Etcher exists. Etcher will give an alert when it's done flashing. Etcher will verify what it wrote. Etcher will predict how much time is remaining.
The raspi imaging utility goes even farther and gives you configuration options. There are many other special purpose flasher utilities like that.
Where dd really shines is in a script, for making empty images of a certain size. But even then... there are tools like truncate to make a sparse file instead.
The only other time I ever need the CLI is for directly creating a compressed image, but that's a somewhat uncommon task for me. And I would not be surprised if one of the GUIs had it by now.
Downloading a giant electron application which bundles Chrome and Xbox joystick drivers just to write to a flash drive feels downright criminal when there's a 200kb program that will do the same job already on your computer.
I personally use dd for such cases as well, but I don't mind downloading a "giant electron application which bundles Chrome and Xbox joystick divers" (I just downloaded Etcher and it takes 40MB on my machine) if that's what it takes to lower the risk of accidentally wiping my hard drive when I want to write an ISO to a USB drive.
Electron makes the problem of flashing totally solved. I have a 1TB SSD for exactly this reason, so I don't have to worry about what's light and what isn't.
I mostly use `gzip -d image.gz | dd of=/dev/sda bs=1M oflag=direct status=progress` to write a compressed disk image to a slow USB stick with progress bar and no disk caching. This avoids the lengthy wait at the end of a typical `cp` for the sync before unplug.
I always assumed that dd was preferred when writing disk images to flash media to ensure each hardware block is written to exactly once, extending the live of the device and increasing performance substantially. I never checked if that's actually a problem with general-purpose file IO, nothing stops the kernel from noticing when a pipe is connected to a block device and configuring an appropriate buffer.
status=progress is also quite handy. You can splice a "| dd status=progress |" in the middle of a pipeline to get a sense of what's happening.
That particular example is an overly silly but I often write
cat file | grep ...
or
cat file | awk '{blah}'
for ad-hoc stuff even though I am fully aware that I could easily avoid the cat simply because having the regex or awk code at the very end makes it easier to read, edit or extend with additional pipeline elements.
The difference with cat is that at least it is looks nicer in some cases. For example, if I want to put the input file at the start of the pipeline, the cat-less version looks weird and confusing.
<file1 somecmd >file2
I wish that shells would just optimize the useless cat behind the scenes, so I could use it without soliciting complaints :)
it's a short walk from shells that optimize "known commands" to signed apps and an ecosystem you can't contribute to unless you've paid the appropriate gatekeeper
I prefer the first way, as you can read it left to right and each step is meaningful. "Take the contents of file, process it through somecmd, then save the result to file2."
I would hope both commands boil down to exactly the same action. Any overhead from invoking cat (isn't that a shell built-in?) should be negligible.
The issue is that the author makes a lot of use of pipes e.g. "|" which has a single page as its buffer and immediatly makes the task pretty much CPU bound. (Because of context switches etc). This is why people use "dd" - It does not need pages/pipes/context switches to stream data in large chunks from a device to a file or vice versa.
Two years ago, I had to clone a disk image from one UEFI notebook over to another identical one. Using gzip with output redirection to the device file (might also have been zcat or cat with gzip) did just not work. After days of unsuccessful attempts, I resorted back to dd with gzip. And believe it or not: Only with dd the disk clone was successful.
I came as far as comparing the image files, which were identical, but I did not have the time to compare the disks. So, this still is a mystery for me.
On a side note: If you ever need to do a full disk clone nowadays, I can only recommend clonezilla. Using FAI (Fully Automatic Installation) would be even better instead of doing image clones, but sometimes, that's not an option.
Good time to mention "sdd" which can handle faulty disks.
It can be set to skip over bad blocks to copy as many healthy blocks as possible, and then go back and retry the bad blocks over and over until they either read successfully or you stop it.
It has a more advanced brother: gnu_ddrescue, ddrescue short. It can nudge and prod disks with severe bad sectors to give up the secrets they hold.
I used it once to recover a 1TB NTFS drive which can't be mounted by Windows. It only failed to recover 12kB in 1TB, which corrupted an MP3 file, which can be easily replaced.
There was a time when "dd" was the only game in town for deep copies on Unix systems. E.g. HPUX in the 1980s.
It always had issues of course: e.g. if the source was a 100MB drive and you dd'ed to a blank 200MB, you'd get a new 100MB drive and all the rest was lost because it was a low level copy. Similarly any bad sectors on the old drive became bad sectors on the new and any bad sectors on the new were simply used as if they were OK.
I don't think so, this command could be significantly slow:
cp myfile.iso /dev/sdb
compared with this one:
dd if=myfile.iso of=/dev/sdb bs=32M
because implementation of cp have a fixed buffer, so if the amount of data is big and the disks fast, using cp you are calling more read() and write() syscalls than necessary, slowing down the copy process.
Last time I researched this, a long time ago: `lz4` is (was?) your best friend.
Oh, but - to follow the submitted article's "rant" - if one wants to use `cat` instead of `dd`, shall there be freedom... Sometimes people like some kind of form-aided "readibility and clarity and method", which leads you to forms like `cat filename | grep` .
That's interesting about the blocksize. I was never impacted by that because I mostly use dd for writing large files, and a large block size always made more sense to me. For example, if I wanted to test writing a 1 GB file to a flash drive, I'd do something like:
I didn't think 'cat' itself could force a sync to disk?
When writing to a USB sd card reader, sometimes the data wouldn't be written entirely without calling 'sync' but dd can perform the sync itself with 'fsync'
I regularly do `cat ~/.ssh/id_rsa.pub | ssh foo@bar tee -a ~/.ssh/authorized_keys` without repercussion. For larger files I usually use rsync as you can resume interrupted transfers
Is there a tool other than dd that would allow me to skip n bytes from a file and then read m bytes? I mean one tool, not a combination of head and tail, which is too elaborate for my taste.
i'm not 100% on this, but i seem to have foggy memories of seeing it used in bizarre ways like this by scripts written to be ultraportable (output of early versions of autotools maybe?).
> dd actually has two jobs: Convert and Copy. A post on comp.unix.misc (incorrectly) claimed that the intended name “cc” was taken by the C compiler, so the letters were shifted in the same way we ended up with a Window system called X. A more likely explanation is given in that thread as pointed out by Paweł and Bruce in the comments: the name, syntax and purpose is almost identical to the JCL “Dataset Definition” command found in 1960s IBM mainframes.
dd found fame about 17 years ago when supposedly low-level copying gui applications would consistently not succeed in some cases for unknown reasons. dd worked where fancier copy applications were hit or miss. So while other methods will work, dd consistently also works. UNIX was designed redundantly intentionally. And I'm pretty sure the U stands for 'useless.'(yeah, jk)
Indeed, most users are better off by launching GNOME Disks, going to the kebab menu, choosing Restore Disk Image & selecting the image and target disk interactively. Or running:
$ gnome-disks --restore-disk-image=/etc/motd
Behind the scenes it calls udisks, which uses polkit to ensure that the user is authorized to write to the target disk.
Overall, the chances of a user typoing and wiping out the wrong disk are greatly reduced. They also get to see progress etc.
I first used it to dual boot windows (98 or NT4 I don't remember) and slackware.
I can't remember specifics, but I dd ed the first 512B to a file and pointed the windows OS loader to that file. The specifics I don't remember are why I wanted to use the windows loader instead of LILO.
The article might educate, but that doesn't appear to be its primary purpose.
The problem with this article and the useless-cat article is that 1) they create a false right vs wrong dichotomy and 2) it gets amplified and used as a tool by the small minded as a citation for "the correct way".
A post on learning your tools is fine. But calling something useless and needlessly adding branches to one's decision tree doesn't empower anyone except jerks.
Stay useless.
Were any of the cited examples harming anything? No they weren't.
One of the great things about `dd` is that you have a lot of control how the input and output files are opened. You can bypass the page cache when reading data by using iflag=direct, which would stop this from happening.