Sammanfattning: Deleted "Many duplicate search tools are commercial" etc because it has nothing to do with emacs -- just link spam.
< [new:gavenkoa:2011-12-14 20:13 UTC]
< Look to list of external tools for finding duplicate code:
< * [http://en.wikipedia.org/wiki/Duplicate_code Wikipedia "duplicate code" article]
< * [http://students.cis.uab.edu/tairasr/clones/literature/#standalone Large list of standalone duplication searching tools]
< [new:gavenkoa:2011-12-14 20:23 UTC]
< Many duplicate search tools are commercial. There are no any good command line open/free source tools. Most of them bundled with or part of another package (like Checkstyle, PMD) or restricted to specific to syntax (only for Java, Python, etc. sources).
The following gives multiple approaches in EmacsLisp code for managing duplicate lines in a file. Try them out by EvaluatingExpressions or by adding to your InitFile. For duplicating lines see CopyFromAbove.
You can try to remove duplicate lines with ReplaceRegexp or `
‘query-replace-regexp’), by replacing
\([^ C-q C-j ]+ C-q C-j \)\1+ with
\1. A newline is entered with a
‘C-q C-j’. The Lisp version of this command would be
(replace-regexp "\\([^\n]+\n\\)\\1+" "\\1")
Since it doesn’t backtrack, there’s no way remove a line with more than one duplicate.
Duplicate line 1 Unique line 1 Duplicate line 1 Duplicate line 1
It will leave duplicate lines.
Duplicate line 1 Unique line 1 Duplicate line 1
Here’s some Lisp to find duplicate lines and keep only the first occurrence by starting each search and replace at the start of the last duplicate –
(defun uniquify-all-lines-region (start end) "Find duplicate lines in region START to END keeping first occurrence." (interactive "*r") (save-excursion (let ((end (copy-marker end))) (while (progn (goto-char start) (re-search-forward "^\\(.*\\)\n\\(\\(.*\n\\)*\\)\\1\n" end t)) (replace-match "\\1\n\\2"))))) (defun uniquify-all-lines-buffer () "Delete duplicate lines in buffer and keep first occurrence." (interactive "*") (uniquify-all-lines-region (point-min) (point-max)))
So for this buffer:
Duplicate line 1 Unique line 1 Duplicate line 1 Unique line 2 Unique line 3 Duplicate line 1 Duplicate line 2 Duplicate line 2 Unique line 4
‘M-x uniquify-all-lines-buffer’ produces:
Duplicate line 1 Unique line 1 Unique line 2 Unique line 3 Duplicate line 2 Unique line 4
Another implementation that adds distinct lines to a temporary list, and checks each line in the list with
‘assoc’ will give the same result as the previous.
(defun uniquify-all-lines-region (start end) "Find duplicate lines in region START to END keeping first occurrence." (interactive "*r") (save-excursion (let ((lines) (end (copy-marker end))) (goto-char start) (while (and (< (point) (marker-position end)) (not (eobp))) (let ((line (buffer-substring-no-properties (line-beginning-position) (line-end-position)))) (if (member line lines) (delete-region (point) (progn (forward-line 1) (point))) (push line lines) (forward-line 1)))))))
The unix utility called
uniq removes duplicate consecutive lines, keeping only one instance. The output of
uniq on the example above is:
Duplicate line 1 Unique line 1 Duplicate line 1 Unique line 2 Unique line 3 Duplicate line 1 Duplicate line 2 Unique line 4
Note that non-consecutive duplication of the first line are not removed.
The unix utility
sort and its
-u argument can give the same result of unique lines as
‘M-x uniquify-all-lines-buffer’ . However, the lines are sorted rather than kept in the order of first appearance.
The output of
sort -u on the example above is:
Duplicate line 1 Duplicate line 2 Unique line 1 Unique line 2 Unique line 3 Unique line 4
‘M-x uniquify-buffer-lines’ will remove identical adjacent lines in the current buffer, similar to what is obtained with the unix
(defun uniquify-region-lines (beg end) "Remove duplicate adjacent lines in region." (interactive "*r") (save-excursion (goto-char beg) (while (re-search-forward "^\\(.*\n\\)\\1+" end t) (replace-match "\\1")))) (defun uniquify-buffer-lines () "Remove duplicate adjacent lines in the current buffer." (interactive) (uniquify-region-lines (point-min) (point-max)))
It is important to note that functions which find duplicate lines don’t always sort lines before looking for dups as this may or may not be what one expects or desires of a particular function.
Where the lines of a file are presorted it can be convenient to use something like this:
(defun find-duplicate-lines (&optional insertp interp) (interactive "i\np") (let ((max-pon (line-number-at-pos (point-max))) (gather-dups)) (while (< (line-number-at-pos) max-pon) (= (forward-line) 0) (let ((this-line (buffer-substring-no-properties (line-beginning-position 1) (line-end-position 1))) (next-line (buffer-substring-no-properties (line-beginning-position 2) (line-end-position 2)))) (when (equal this-line next-line) (setq gather-dups (cons this-line gather-dups))))) (if (or insertp interp) (save-excursion (new-line) (princ gather-dups (current-buffer))) gather-dups)))
This function, while inefficient (note cons in tail of while form) is quite handy for locating duplicates before removing them, i.e. situations of type:
‘find-duplicate-lines’ by comparing its result list with one or more of the list comparison procedures
‘intersection’, etc. from the CL package (require ‘cl).