Innehållsförteckning RecentChanges News ElispArea HowTo Problems Suggestions

DuplicateLines

Last edit

Sammanfattning: Deleted "Many duplicate search tools are commercial" etc because it has nothing to do with emacs -- just link spam.

Deleted:

< [new:gavenkoa:2011-12-14 20:13 UTC]
< Look to list of external tools for finding duplicate code:
< * [http://en.wikipedia.org/wiki/Duplicate_code Wikipedia "duplicate code" article]
< * [http://students.cis.uab.edu/tairasr/clones/literature/#standalone Large list of standalone duplication searching tools]
< [new:gavenkoa:2011-12-14 20:23 UTC]
< Many duplicate search tools are commercial. There are no any good command line open/free source tools. Most of them bundled with or part of another package (like Checkstyle, PMD) or restricted to specific to syntax (only for Java, Python, etc. sources).
< ----


The following gives multiple approaches in EmacsLisp code for managing duplicate lines in a file. Try them out by EvaluatingExpressions or by adding to your InitFile. For duplicating lines see CopyFromAbove.

Interactive search and replace won't work

You can try to remove duplicate lines with ReplaceRegexp or `C-M-%’ (‘query-replace-regexp’), by replacing \([^ C-q C-j ]+ C-q C-j \)\1+ with \1. A newline is entered with a ‘C-q C-j’. The Lisp version of this command would be

  (replace-regexp "\\([^\n]+\n\\)\\1+" "\\1")

Since it doesn’t backtrack, there’s no way remove a line with more than one duplicate.

Duplicate line 1
Unique line 1
Duplicate line 1
Duplicate line 1

It will leave duplicate lines.

Duplicate line 1
Unique line 1
Duplicate line 1

Search and replace in Lisp that works

Here’s some Lisp to find duplicate lines and keep only the first occurrence by starting each search and replace at the start of the last duplicate – (goto-char start).

  (defun uniquify-all-lines-region (start end)
    "Find duplicate lines in region START to END keeping first occurrence."
    (interactive "*r")
    (save-excursion
      (let ((end (copy-marker end)))
        (while
            (progn
              (goto-char start)
              (re-search-forward "^\\(.*\\)\n\\(\\(.*\n\\)*\\)\\1\n" end t))
          (replace-match "\\1\n\\2")))))
  
  (defun uniquify-all-lines-buffer ()
    "Delete duplicate lines in buffer and keep first occurrence."
    (interactive "*")
    (uniquify-all-lines-region (point-min) (point-max)))

So for this buffer:

Duplicate line 1
Unique line 1


Duplicate line 1
Unique line 2
Unique line 3

Duplicate line 1


Duplicate line 2
Duplicate line 2
Unique line 4

running ‘M-x uniquify-all-lines-buffer’ produces:

Duplicate line 1
Unique line 1

Unique line 2
Unique line 3
Duplicate line 2
Unique line 4

A Lisp command using a list containing each line

Another implementation that adds distinct lines to a temporary list, and checks each line in the list with ‘assoc’ will give the same result as the previous.

  (defun uniquify-all-lines-region (start end)
    "Find duplicate lines in region START to END keeping first occurrence."
    (interactive "*r")
    (save-excursion
      (let ((lines) (end (copy-marker end)))
        (goto-char start)
        (while (and (< (point) (marker-position end))
                    (not (eobp)))
          (let ((line (buffer-substring-no-properties
                       (line-beginning-position) (line-end-position))))
            (if (member line lines)
                (delete-region (point) (progn (forward-line 1) (point)))
              (push line lines)
              (forward-line 1)))))))

The uniq command with consecutive lines

The unix utility called uniq removes duplicate consecutive lines, keeping only one instance. The output of uniq on the example above is:

Duplicate line 1
Unique line 1

Duplicate line 1
Unique line 2
Unique line 3

Duplicate line 1

Duplicate line 2
Unique line 4

Note that non-consecutive duplication of the first line are not removed.

The sort -u command for non-consecutive duplicates

The unix utility sort and its -u argument can give the same result of unique lines as ‘M-x uniquify-all-lines-buffer’ . However, the lines are sorted rather than kept in the order of first appearance.

The output of sort -u on the example above is:

Duplicate line 1
Duplicate line 2
Unique line 1
Unique line 2
Unique line 3
Unique line 4

Lisp commands removing consecutive duplicates

The command ‘M-x uniquify-buffer-lines’ will remove identical adjacent lines in the current buffer, similar to what is obtained with the unix uniq command.

  (defun uniquify-region-lines (beg end)
    "Remove duplicate adjacent lines in region."
    (interactive "*r")
    (save-excursion
      (goto-char beg)
      (while (re-search-forward "^\\(.*\n\\)\\1+" end t)
        (replace-match "\\1"))))
  
  (defun uniquify-buffer-lines ()
    "Remove duplicate adjacent lines in the current buffer."
    (interactive)
    (uniquify-region-lines (point-min) (point-max)))

It is important to note that functions which find duplicate lines don’t always sort lines before looking for dups as this may or may not be what one expects or desires of a particular function.

Lisp command to retrieve duplicates

Where the lines of a file are presorted it can be convenient to use something like this:

  (defun find-duplicate-lines (&optional insertp interp)
    (interactive "i\np")
    (let ((max-pon (line-number-at-pos (point-max)))
	  (gather-dups))
      (while (< (line-number-at-pos) max-pon) (= (forward-line) 0)
	     (let ((this-line (buffer-substring-no-properties (line-beginning-position 1) (line-end-position 1)))
		   (next-line (buffer-substring-no-properties (line-beginning-position 2) (line-end-position 2))))
	       (when  (equal this-line next-line)  (setq gather-dups (cons this-line gather-dups)))))
      (if (or insertp interp)
	  (save-excursion (new-line) (princ gather-dups (current-buffer)))
	gather-dups)))

This function, while inefficient (note cons in tail of while form) is quite handy for locating duplicates before removing them, i.e. situations of type: ‘uniquify-maybe’. Extend ‘find-duplicate-lines’ by comparing its result list with one or more of the list comparison procedures ‘set-difference’, ‘union’, ‘intersection’, etc. from the CL package (require ‘cl).


CategoryEditing