過去我曾多次在這裡提出類似的問題,並取得了巨大的成功,但現在我的需求略有變化,我正在努力獲得我正在尋找的確切輸出。
我想比較2相似的分隔文件,但它們將具有不同的行數和一些重複項。這些文件將具有相同的標頭。
file1.txt
mem_id date time building
aa1 bb1 cc1 dd1
aa2 bb2 cc2 dd2
aa3 bb3 ccx3 dd3
aa4 bb4 cc4 dd4
aa5 bb5 cc5 dd5
file2.txt
mem_id date time building
aa1 bby1 cc1 ddy1
aa2 bb2 cc2 dd2
aa3 bb3 cc3 dd3
aa4 bb4 cc4 dd4
aa4 bb4a cc4a dd4a
你會發現有 4 個不同點:
1- File2,mem_id aa1 在“日期”和“建築物”欄中都有“y”
2- File1,mem_id aa3 在“時間”欄中有一個“x”
3- File1,有一個 mem_id aa5
4- File2,mem_id aa4 有 2 個條目
我想運行一個腳本來僅輸出兩個文件之間的差異(跳過相同的行)。我嘗試過的所有內容都會掛在重複或跳過的行上,從而弄亂整個文件的輸出。如果所有行都匹配,則以下程式碼可以正常運行:
current_code
awk -F ',' 'BEGIN {IGNORECASE = 1} NR==1 {for (i=1; i<=NF; i++) header[i] = $i}NR==FNR {for (i=1; i<=NF; i++) {A[i,NR] = $i} next}{ for (i=1; i<=NF; i++) if (A[i,FNR] != $i) print header[1]"#-"$1": " header[i] "- " ARGV[1] " value= ", A[i,FNR]" / " ARGV[2] " value= "$i}'
desired_output.txt
Mem_id#-aa1 : date- file1.txt value = bb1 / file2.txt value= bby1
Mem_id#-aa1 : building- file1.txt value = dd1 / file2.txt value= ddy1
Mem_id#-aa3 : time- file1.txt value = ccx3 / file2.txt value= dd3
Mem_id#-aa4 : date- file1.txt value = / file2.txt value= bb4a
Mem_id#-aa4 : time- file1.txt value = / file2.txt value= cc4a
Mem_id#-aa4 : building- file1.txt value = / file2.txt value= dd4a
Mem_id#-aa5 : date- file1.txt value = bb5 / file2.txt value=
Mem_id#-aa5 : time- file1.txt value = cc5 / file2.txt value=
Mem_id#-aa5 : building- file1.txt value = dd5 / file2.txt value=
答案1
以下 python 程式應該執行您想要的操作,或非常接近它的操作。
第三行
desired_output.txt
似乎是錯的:Mem_id#-aa3 : time- file1.txt value = ccx3 / file2.txt value= dd3
dd3 should probably be
cc3`除此之外,程式的輸出除了空格之外都匹配,這在範例輸出中似乎有點不規則。
輸入被認為是按鍵(memid)排序的
- 程式在嘗試同步時預設緩衝 4 行 (max_diff + 1)。如果該緩衝區中的鍵中沒有一個與「目前」鍵匹配,則 vv 均被視為不匹配並列印,並嘗試下一對。如果找到某個鍵,則將其他緩衝區中的不匹配項或先輸出。
當第一行和第二行具有相同的 memid 兩次(或更多)時,範例輸入對預期的行為有一些限制。
在
output()
我嘗試匹配任何行並彈出所有匹配(從左到右)。因此,在同一 memid 中匹配行的順序並不重要。如果左側或右側或兩者都為空,則列印很容易(特別是當兩者都為空時)。對於其餘的,我從左到右匹配剩餘的每一行。fmt
中的字串決定line_out()
輸出,您可以自由更改/重新排序。
#! /usr/bin/env python
# coding: utf-8
# http://unix.stackexchange.com/q/161913/33055
from __future__ import print_function
from collections import OrderedDict
from logging import debug
import sys
class RowBuffer:
def __init__(self, file_name, delim=None, max_diff=3):
"""delim is the character that is used for splitting input.
None->whitespace
"""
self._verbose = 0
self._file_name = file_name
self._fp = open(self._file_name)
self._delim = delim
self._max_diff = max_diff
self._head = self._fp.readline().split(delim)
# the buffer consists of a maximum of max_diff entries
# the keys are the first items of a row, the value a list
# of all other items on that row
self._buffer = OrderedDict()
self.fill_buffer()
def compare(self, rb):
"""check if self._buffer"""
if self._head != rb._head:
print('headings differ:\n {}\n {}'.format(
self._head, rb._head))
while self._buffer:
l = self.get()
try:
r = rb.get()
except KeyError:
debug('only left %s', l[0])
self.output(l, None, rb)
break
if l[0] == r[0]:
debug('compare vals %s', l[0])
self.output(l, r, rb)
continue
if l[0] in rb:
# left key in right, but not at top
# output right until top keys are same
while l[0] != r[0]:
debug('only right %s', r[0])
self.output(None, r, rb)
r = rb.get()
self.output(l, r, rb)
continue
if r[0] in self:
# right key in left, but not at top
# output left until top keys are same
while l[0] != r[0]:
debug('only left %s', l[0])
self.output(l, None, rb)
l = self.get()
self.output(l, r, rb)
continue
# neither found: output both
debug('neither left in right nor vv %s %s', l[0], r[0])
self.output(l, None, rb)
self.output(None, r, rb)
while rb._buffer: # remaining in right file
r = rb.get()
debug('only right %s', r[0])
self.output(None, r, rb)
def output(self, l, r, right):
fmt1 = '{col0_header}#-{col0_value} : {col_header}- ' \
'{left_file_name} value = {left_value} / ' \
'{right_file_name} value= {right_value}'
d = dict(
col0_header=self._head[0],
left_file_name=self._file_name,
right_file_name=right._file_name,
)
if l is not None and r is not None:
# one or more values on both sides, compare all lines on the
# left with all on the right remove any matching pairs
match = {} # left index to right index
for lidx, lv in enumerate(l[1]):
for ridx, rv in enumerate(r[1]):
if lv == rv:
if lidx not in match:
match[lidx] = ridx
# pop from back of list, not invalidate index
for lidx in sorted(match, reverse=True):
l[1].pop(lidx)
for ridx in sorted(match.values(), reverse=True):
r[1].pop(lidx)
if r is None or not r[1]:
for lv in l[1]:
for idx, k in enumerate(self._head[1:]):
self.line_out(d, col0_value=l[0], col_header=k,
left_value=lv[idx], right_value=' ')
return
if l is None or not l[1]:
for rv in r[1]:
for idx, k in enumerate(self._head[1:]):
self.line_out(d, col0_value=l[0], col_header=k,
left_value=' ', right_value=rv[idx])
return
# print non matching
for lv in l[1]:
for rv in r[1]:
for idx, k in enumerate(self._head[1:]):
if lv[idx] == rv[idx]:
continue # same value
self.line_out(d, col0_value=l[0], col_header=k,
left_value=lv[idx], right_value=rv[idx])
def line_out(self, d, **kw):
# manipulate and print output
# the fields of the format string can be arbitrarily arranged
# as long as the field names (between {} match)
fmt = '{col0_header}#-{col0_value} : {col_header}- ' \
'{left_file_name} value = {left_value} / ' \
'{right_file_name} value= {right_value}'
d1 = d.copy()
d1.update(kw)
s = fmt.format(**d1)
# s = s.rstrip()
s = s[0].upper() + s[1:] # sample output doesn't match input
print(s)
def get(self):
item = self._buffer.popitem(last=False)
self.fill_buffer()
return item
def fill_buffer(self):
if self._fp is None:
return
while len(self._buffer) < self._max_diff:
row = self._fp.readline().split(self._delim)
if not row:
self._fp.close()
self._fp = None
return
entry = self._buffer.setdefault(row[0], [])
entry.append(row[1:])
def __contains__(self, key):
self.fill_buffer()
return key in self._buffer
rb1 = RowBuffer(sys.argv[1])
rb2 = RowBuffer(sys.argv[2])
rb1.compare(rb2)
答案2
這是對您的問題的(遠非優雅的)部分解決方案。它使用第一列作為 id 列(它不必是第一列,但你絕對必須有一個)並引入第三個維度suffix
來儲存同一鍵的多次出現。最後它嘗試找到文件 2 中那些在文件 1 中沒有找到的鍵。
BEGIN {
IGNORECASE = 1
}
NR==1 {
for (i = 1; i <= NF; i++)
header[i] = $i
suffix = 0
previous_key=""
}
NR==FNR {
if ($1 == previous_key) {
suffix = suffix + 1
max_suffix[$1] = suffix
} else
suffix = 0
for (i = 1; i <= NF; i++) {
A[$1,suffix,i] = $i
}
key_count[$1] = key_count[$1] + 1
previous_key = $1
next
}
{
if ($1 == previous_key)
suffix = suffix + 1
else
suffix = 0
previous_key = $1
if (A[$1,suffix,1] != "") {
for (i = 2; i <= NF; i++)
if (A[$1,suffix,i] != $i) {
print header[1]"#-"$1": " header[i] "- " ARGV[1] " value= ", A[$1,suffix,i]" / " ARGV[2] " value= "$i
}
key_count[$1] = key_count[$1] - 1
}
else
for (i = 2; i <= NF; i++)
print header[1]"#-"$1": " header[i] "- " ARGV[1] " value= ", " / " ARGV[2] " value= "$i
}
END {
for (missing_key in key_count)
if (key_count[missing_key] > 0) {
for (suffix = max_suffix[missing_key] - key_count[missing_key] + 1; suffix <= C[missing_key]; suffix++)
for (i = 2; i <= NF; i++)
print header[1]"#-"missing_key": " header[i] "- " ARGV[1] " value= ", A[missing_key,suffix,i] " / " ARGV[2] " value= "
}
}
有一個警告:文件 2 中的不匹配條目始終打印在末尾,並且不根據文件中的位置進行排序。此外,這些行的排序是任意的。我想這可以透過將結果傳遞到命令中來實現sort
。