I have backed up my system to an external ximeta drive using "dd" and the well-known linux live cd distribution, Knoppix to boot from. Below are the steps in brief:
-
Boot from the live cdrom distribution.
Switch to root.
Make sure NO partitions are mounted from the source hard drive.
Mount the external HD.
# mount -t vfat /dev/sda1 /mnt/sda1Backup the drive.
# dd if=/dev/hda conv=sync,noerror bs=64K | gzip -c > /mnt/sda1/hda.img.gz
"dd" is the command to make a bit-by-bit copy of "if=/dev/hda" as the "Input File" to "of=/mnt/sda1/hda.img.gz" as the "Output File". Everything from the partition will go into an "Output File" named "hda.img.gz". "conv=sync,noerror" tells dd that if it can't read a block due to a read error, then it should at least write something to its output of the correct length. Even if your hard disk exhibits no errors, remember that dd will read every single block, including any blocks which the OS avoids using because it has marked them as bad. "bs=64K" is the block size of 64x1024 Bytes. Using this large of block size speeds up the copying process. The output of dd is then piped through gzip to compress it.
To restore your system:# gunzip -c /mnt/sda1/hda.img.gz | dd of=/dev/hda conv=sync,noerror bs=64K
NOTE: I've had much success leaving out "conv=sync,noerror" during restore.
Store extra information about the drive geometry necessary in order to interpret the partition table stored within the image. The most important of which is the cylinder size.# fdisk -l /dev/hda > /mnt/sda1/hda_fdisk.info
Notes:
One of the disadvantages of the dd method over software specifically designed for the job such as Ghost or partimage is that dd will store the entire partition, including blocks not currently used to store files, whereas the likes of Ghost understand the filesystem and don't store these unallocated blocks. The overhead isn't too bad as long as you compress the image and the unallocated blocks have low entropy. In general this will not be the case because the emtpy blocks contain random junk from bygone files. To rectify this, it's best to blank all unused blocks before making the image. After doing that, the unallocated blocks will contain mostly zeros and will therefore compress down to almost nothing.
Mount the partition, then create a file of zeros which fills the entire disk, then delete it again.
# dd if=/dev/zero of=/tmp/delete.me bs=8M; rm delete.me
References:
- sandip's blog
- Login or register to post comments
Comments
Like many people here - I was having issues with corrupt restores.
I was restoring from a USB / SD Card. Using "sync" I believe what was happening is that gzip couldn't provide the 64k fast enough so it was, as the documentation suggests, filling the gaps with NULL/ZERO and causing corruption.
REMOVE sync,noerror, but it is safe to leave the bs=64k in there, as you'll get much better performance. I believe now, what happens is that dd waits for each 64K chunk rather than filling the gaps. So on my hard drive image, it started out at around 18MB/s and then as it got to the blank portions of the drive where compression was much higher - it increased to 50 MB/s.
I was trying backup my Windows system using this same strategy. To prepare the Windows partition, I created a 10G /dev/zero file in Linux, copied over to Windows, and kept coping it over and over until I filled up the disk. I then deleted them, booted a thumb drive to Linux, and tried to DD the Windows partition.
The problem I had was I couldn't get the image to be any smaller than the full 750G of the disk, piping through gzip.
I'm curious if the Windows partition doesn't present any redundancy to the Linux OS to take advantage of. Does anyone know? Does NTFS do some 'clever' Windows thing where the disk is actually discontinuous outside of Windows?
i know this is an old post but block sizes also use that exact amount of memory during the dd process.
I use 5G block sizes and you must have more that 5G of Ram available etc
this speeds up the process greatly. like 10x
Excellent Blog. Thank you so much for taking your time to write this.
Hello everybody,
I read this page twice before I started using dd.
I have a HDD with 2 partitions: Windows 7 (NTFS format) and CentOS6.3 (ext4 format) and I backup all the disk with this command:
"dd if=/dev/sda conv=sync,noerror bs=64K | gzip -c > /*/sda.img.gz"
and after the backup was finished, I got no errors.
I restored the image on another SSD and I got this error/worning:
{{{
dd: writing to `/dev/sda': No space left on device
234441649+0 records in
234441648+0 records out
120034123776 bytes (120 GB) copied, 2192.83 s, 54.7 MB/s
}}}
The SSD's type are the same, exactly the same.
The information on the SSD is around 20GB, and on the first SSD, that one which I backuped, has 5GB free space.
I used GParted soft for checking the SSD whit a Clonezilla-SysRescCD.
For ext4 partition I got:
{
e2label no such file or directory while trying to open /dev/sda3
Couldn't find valid filesystem superblock.
Couldn't find valid filesystem superblock.
dumpe2fs 1.41.10 (10-Feb-2009)
dumpe2fs: No such file or
directory while trying to open /dev/md126p1
Unable to read the contents of this file system!
Because of this some operations may be unavailable.
The following list of software packages is required for ext3 file system support: e2fsprogs.
}
For NTFS partition I got:
{
The device doesn't exist.
ntfsresize v2.0.0 (libntfs 10:0:0)
Error(2):Failed to check /dev/sda/ mount state: No such file or directory
Probably /etc/mtab is missing. It is too risky to continue. You might try another Linux distro.
Unable to read the file system contents of this file system. Because of this some operations may be unavailable. The causer might might a missing software package. The following list of support packages is required for ntfs file system support.
ntfsprogs /ntfs-3g
}
Do you think this is because I am using an old distribution of SysRescCD?
Do you have any idea what is the problem?
All the best,
Remus.
Hi there,
just wanted to draw your attention to the fact that zipping the image is quite demanding with respect to cpu power. If you have enough space its better to avoid this - in my case using compression was around 15 times slower....
this would work like:
# dd if=/dev/hda conv=sync,noerror bs=64K of=/mnt/sda1/hda.img
But if you do this over slow connections or have not much space for an image its of course fine to compress...
Hi, just want to do a miniature correction. Making the file delete.me in /tmp is a very bad idea.
/tmp is actually of the tmpfs kind. Meaning that the /tmp partition is actually your RAM.
Essentially, you'll be filling up your ram (let's agree that's never a good idea). And will wind up with none of it usable.
On top of that, it's quite possible you have other partitions!
So essentially, one would do
sudo mount | grep /dev/hdato find where the partition in question is mounted, then in that location... (let's assume it's /home/mystorage)sudo dd if=/dev/zero of=/home/mystorage/delete.me && sudo rm /home/mystorage/delete.meand not touch that specific partition for a while.First of all, thanks for the article.
Now about the disadvantages of brute imaging by dd. I'd actually consider it an advantage, cause commercial imaging software seems to think too high of its own intelligence and won't give an option of making a "dumb" image of a partition if it "knows" this FS. There are numerous occasions when I'd like to preserve everything verbatim, including errors. Well, come to think of it, there was Norton Ghost 8 bootable that had an option "no compression". I think that implied full dumb copy without smarting off, but I may be wrong. Thank god we have dd in Linux, cause I don't know how to deal with this in the Windows world (which I do have to deal with).
Incidentally, if someone of you knows an _obedient_ commercial imaging tool with this kind of control (preferably with GUI) available on boot media, I would very much appreciate your sharing such info.
Thanks.
Hi,
I am little hesitant to use dd command as many scare to lose the data if used wrongly. I want to ask few questions before i could use that command.
My hard disk has crashed but i can open it in rescue mode.
I want to take a full backup before i can re-install a completely new OS.
I have three partitions in it on /dev/cciss/c0p0d1, /dev/cciss/c0d0p2, /dev/cciss/c0d0p3.
The hard disk is mounted on /dev/cciss/codo
my external hard disk /dev/sda is mounted on /mnt/sda2 in root directory.
Do i have to unmount it before using dd command, also, my extHD is 500GB, with two parts of 250GB each.
If i use a dd command fora 90GB hard disk, will it occupy whole 250GB.
Please help me in this regard.
Thanks
dd is best done with drive unmounted so nothing can be written to the file while backup is being performed.
As mentioned above, fill up the drive and then delete the file so the compressed image does not take up much space.
dd if=/dev/zero of=/tmp/delete.me bs=8M; rm delete.medd if=/dev/hda conv=sync,noerror bs=64K | gzip -c > /mnt/sda1/hda.img.gz
Kind of late, but;
The correct response should have been,
If you want to "zero out" the "empty" portion of your partitions, find where the three partitions are mounted, then
While they're mounted,
dd if=/dev/zero of=/[some directory in partition you want to zero out]/delete.me bs=8M; rm /[some directory in partition you want to zero out]/delete.me
Repeat for each partition you want to zero out.
if the entire disk is 90 GB, you probably don't want to zero it out(allows you to possibly recover deleted files)
then after you unmount or remount read only the partitions (all three),
dd if=/dev/cciss/codo conv=sync,noerror bs=64K | gzip -c > /mnt/sda2/hda.img.gz
it will create a file called hda.img.gz so it won't take up the entire 250GB partition.
Quote
I have three partitions in it on /dev/cciss/c0p0d1, /dev/cciss/c0d0p2, /dev/cciss/c0d0p3.
The hard disk is mounted on /dev/cciss/codo
my external hard disk /dev/sda is mounted on /mnt/sda2 in root directory.
Do i have to unmount it before using dd command, also, my extHD is 500GB, with two parts of 250GB each.
If i use a dd command fora 90GB hard disk, will it occupy whole 250GB.
end quote
I used this cat /dev/sda | gzip -9 > hdbackup.img.gz
in a shell
and want to do this too
dd if=/dev/zero of=/tmp/delete.me bs=8M; rm delete.me
and I dont know for sure what to do (do you use mount hdbackup.img.gz /image -o loop)
You can also use the cat command and pipe it through gzip with the "-9" option (for maximum compression) and then redirect the output to your image file:
cat /dev/sda | gzip -9 > hdbackup.img.gz
It works for me, but I have only been using Linux for less than a year (I like it better than windows now, witch I have been using all my life). So don't tell me if you have any problems with it...
-- A 13 year old anonymous Linux geek
Would `gzip -cd hdbackup.img.gz > /dev/sda` be a good way to restore the image, as well? :-D
Good Job,
Keep learning and know that trying something new and bringing your youthful experience in doing something a different/simpler way
is a great thing !!
I'm trying to restore an image back to the original disk, and unfortunately it's not working. It's a dual boot box, XP and Gentoo, and what I get after the restore is a bogus partition table according to my boot disk. I can see two partitions, the windows and extended like so:
******************************
fdisk -l /dev/sdc
Warning: invalid flag 0x0000 of partition table 5 will be corrected by w(rite)
Disk /dev/sdc: 160.0 GB, 160041885696 bytes
129 heads, 4 sectors/track, 605778 cylinders
Units = cylinders of 516 * 512 = 264192 bytes
Disk identifier: 0x8ff57ec4
Device Boot Start End Blocks Id System
/dev/sdc1 * 1 406425 104857648 7 HPFS/NTFS
/dev/sdc2 406426 605778 51433074 5 Extended
******************************
Here's what I actually backed up:
rabbit scoutBackup # cat scoutHdFdisk.info
Disk /dev/sdc: 160.0 GB, 160041885696 bytes
129 heads, 4 sectors/track, 605778 cylinders
Units = cylinders of 516 * 512 = 264192 bytes
Disk identifier: 0x8ff57ec4
Device Boot Start End Blocks Id System
/dev/sdc1 * 1 406425 104857648 7 HPFS/NTFS
/dev/sdc2 406426 605778 51433074 5 Extended
/dev/sdc5 * 406426 406804 97780 83 Linux
/dev/sdc6 406805 414556 2000014 83 Linux
/dev/sdc7 414557 605778 49335274 83 Linux
rabbit scoutBackup #
Looks like it has something to do with restoring the extended partitions. Any tips on how to fix this?
I also tried manually partitioning the hard drive just like my backed up fdisk -l info, and still no joy on booting up. I'm re-restoring it from the image, and I'll take a fresh stab at it.
/me sighs....
Restoring the image with this command worked for me:
gzip -cd /mnt/sda1/hda.img.gz | dd of=/dev/hda
Just tried this on a partition last night. Using a blocksize of 64K to create the image works, but doing so to restore the image did not work. I had to restore it using the same command you posted (thanks for that). Any idea why trying this with a bs of 64K does not work? Is it possible that dd is faster than gzip and is asking to read 64K from stdin and when it doesn't get that because gzip is too slow, it pads with 0's, causing corruption? Restoring this way is REALLY slow.
Good to know you finally got it to work.
hello , i am new to linux. i was woudering why we are using bs=64K not 4K (4096/1024) or somthing like that.
plz explain. thx for the article.
Using large block size speeds up the copying process.
Thanks for the reminder that you can use dd to do a disk backup / restore. Sometimes we forget that the simple low-level tools are still there.
One other alternative I wanted to mention is clonezilla (http://clonezilla.org/). The documentation is a bit lacking but it's a very powerful tool that duplicates and extends the norton ghost functionality. In the past I've set up a PXE-boot clonezilla config so you can remote-boot a machine and clone the OS directly onto it over the network without using any removable media.
Just want to say this post was a great help, I also found another post at this address http://rob-ward.co.uk/blog/ddb
And again great post, saved me loads of time :-)
the method for restoring the hdd in the article doesn't work for me.
# gunzip -c /mnt/sda1/hda.img.gz | dd of=/dev/hda conv=sync,noerror bs=64Kalthough my source image was correct, it wasn't transferred properly to the hdd. i inspected the two images, and realized that all the data was being transferred, but it was being regularly padded with a chunk of zeros every so often. Here's what I think is happening: either the pipe fills up, gzip runs too slow, or (since i'm transferring the image over the network) the transfer time is too slow. This means dd will sometimes get a chunk of data less than the 64K blocksize, and since sync is enabled, it will pad the rest of the 64K with zeros. bad!
i would recommend just leaving off sync and noerror (since they have to do with read errors, which you really shouldn't get because you're reading the image)
has anyone else run into this problem? what are your thoughts?
Sometimes on restoring a dd image for Windows partitions, the drive would just hang with a blinking cursor on boot. I wonder if this is because dd pads zeroes to the boot sector. I will try your suggestion to remove the conv=sync,noerror. Thanks!
In my case this happened because the windows bootmgr (or ntldr on older windows versions) was on a different hard drive than the Windows boot partition and that hard drive died (it was a small 512MB internal flash drive). By default Windows will try to install the system partition (which contains the bootmgr) on the first drive it sees in the system.
In your case I'll guess that you had a single drive and windows created a small partition at the front of the drive for the system files (bootmgr) and your windows installation (boot partition) was on a different partition. I would guess maybe you didn't back up the entire drive and only backed up the Windows boot partition?
Since I didn't have a backup of my system partition I solved this by re-installing windows and then using dd to copy my windows boot partition image over the newly installed boot partition and the system recovered.
However you ended up backing up and restoring your drive the problem is that the MBR can't find the partition that the bootmgr resides on. Just as an FYI... for Windows installs the following can be true: you point your BIOS to a hard drive you want to boot from (that has an MBR pointing to the bootmgr). When the BIOS gives control to this MBR it then needs to know about the location of the bootmgr (or grub if running linux). The MBR launches the bootmgr which needs to know about all of your Windows installations.... the bootmgr will ask which version of Windows you want to boot (assuming you have more than one) and then will pass control to Windows so it can start.
I've had much success leaving out "conv=sync,noerror" during restore.
Wondering if I should just use bs=512 instead of bs=64k or bs=1M. Sure it's lots slower, but if it means actually having a working image when I need it, ... ???
I just tried a backup and restore using your instructions. I was getting a "no space left on device" error on restore and found this to work instead:
# gzip -cd /mnt/sda1/hda.img.gz | dd of=/dev/hda
Thanks for the article, works great apart from that!
That looked to be an error with the command option for gzip. Thanks, it has been corrected.
If the sector size of partitions are known, you can extract partition images from the main image via:
# fdisk -l -u /dev/sdaOutputs as:
So to create an image of the second partition (sda2) from existing image of sda...
# dd if=/path/to/sda.img of=/tmp/sda2.img bs=512 skip=16771860 count=$[75489434-16771860]I just came across this tutorial today, and I noticed what I think is an error in the above post: shouldn't the "skip" parameter be set to Start-1? In other words, for the above example, it should be skip=$[16771860-1], because skip starts after the block you specify. Is there anyone who would confirm this?
sector count start at zero not at one, so the above should be fine.
this is for only if it doesn't copy that sector you can use dd to copy that sector to a new file:
# dd if=/dev/sda of=/tmp/sda2-sector.img bs=512 skip=$[16771860-1] count=1and then cat the two together:
# cat /tmp/sda2-sector.img /tmp/sda2.img > /tmp/sda2-complete.imgremember this is ONLY for if it doesn't include the first sector of the partition!!!
Copy /dev/sda to /dev/sdb:
# dd if=/dev/sda of=/dev/sdb conv=noerror,sync- /dev/sda: Source disk
- /dev/sdb: Target disk
- sync: Use synchronized I/O for data and metadata
- noerror: Continue copy operation after read errors
This stores the first 512 bytes of the disk (contianing the MBR and the primary partition info - i.e. the first four primary entries) into the file "mbr.img".
To restore (be careful - this could destroy your existing partition table and with it access to all data on the disk):
If you only want to restore the actual MBR code and not the primary partition table entries, just restore the first 448 bytes of the MBR:
# dd if=/dev/hda conv=sync,noerror bs=64K | gzip -c > /mnt/sda1/hda.img.gzAlso I think that should be a lowercase 'k' in 'bs=64k', dd doesn't seem to like the capitalization.
> Also I think that should be a lowercase 'k' in 'bs=64k', dd doesn't seem to like the capitalization.
That's true for the BSD dd (at least the one that comes with Mac OS X 10.3). GNU dd, on the other hand, uses k=1000 and K=1024.
That's why it's a good idea to consult local man pages (or at least distribution-specific man pages).
- Chris
1k=1000bytes
1K=1024bytes
Use factors of 512 to do your copying, since 512 bytes is the sector size used on your HDD.
1k = 1000 unspecified units of measurement
1K = 1 Kelvin = 274.15°C
1kb = 1000 bits
1kB = 1024 bytes
1kB is 1000 Bytes SI notation, 1024 was back in the days because it was so simular.
1KiB is now 1024 Bytes - has been for a while already.
Alternatively you are seeing foke using KB instead of KiB to speak of 1024, ignoring the Kelvin facts.
Plz let us just use SI notation or the alternative created notation, this is what the distro's and other operating systems are using more and more.
-
ontopic: thanks for a lesson in dd
1 Kelvin == -272.15 deg C
In the last example, shouldn't "448" be "446" instead?
The 512byte MBR sector is actually two parts:
1. The first 446 bytes of the sector are the grub stage1 bootloader or the windows bootloader.
2. The last 64 bytes is where the partition table is stored.
So it does not really matter whether it is 448 or 446 since this area of 2 bytes contains a constant that doesn’t change.
The MBR consists of _three_ parts, not two:
1. The first 446 bytes contain the boot loader.
2. The next 64 bytes contain the partition table (4 entries, each 16 bytes).
3. The last 2 bytes contain the identifier 0xAA55 (or 0x55AA, I forgot).
To copy the boot loader only, one *must* use "bs=446". To copy the partition table only, one must use "bs=1 skip=446 count=64".