According to the Emacs manual:

On most operating systems, rewriting a file automatically destroys all record of what the file used to contain. Thus, saving a file from Emacs throws away the old contents of the file – or it would, except that Emacs carefully copies the old contents to another file, called the “backup” file, before actually saving.

How backups are operated in Emacs is configurable to some degree:

    (setq backup-by-copying t   ; don't clobber symlinks
          version-control t     ; use versioned backups
          delete-old-versions t
          kept-new-versions 6
          kept-old-versions 2)

Delete backup files

          find -name "*~" | xargs rm
If the file names contain whitespace, it is necessary to use one of:
          find -name "*~" -print0 | xargs -0 rm
          find -name "*~" -exec rm {} \;
          find -name "*~" -delete

Hiding backup files (MS Windows)

When viewing a source code directory in Windows Explorer, the littering by backup files can quickly become distracting. With the subsequent code new backup files will be hidden.

(when (eq system-type 'windows-nt)
  (defadvice backup-buffer (after my-backup-make-hidden activate)
    (let ((backup-file-name (make-backup-file-name (buffer-file-name))))
      (when (file-exists-p backup-file-name)
	(call-process "attrib.exe" nil nil nil "+I" "+H" backup-file-name)))))

See also AutoSave, BackupDirectory and BackupEachSave.