使用 fontspec 的 Node 渲染器時，Devanagari UTF-8 文字擷取程式碼不會產生 lua 錯誤，但由於 Node 渲染器中可能存在錯誤，因此仍然無法產生正確的結果。我已將其作為一個單獨的問題提出：LuaTeX：Devanagari字形順序在tex的內部節點清單中顛倒了，如何在遍歷字形節點時恢復正確的順序？
在嘗試從TeX 框中提取UTF-8 文本的不同技術時，我遇到了兩種使用fontspec 的Node 渲染器提取梵文文本時不會產生任何lua 錯誤的技術，但在使用fontspec 的HarfBuzz 渲染器（Renderer = Harfbuzz，渲染器=OpenType）。
這兩種技術的詳細資訊如下：技術1（使用麥可-H21的get_unicode函數）和這裡：技術2（僅適用unicode.utf8.char於複雜字形的組成部分）。我嘗試了多種梵文字體，所有結果都相同。
兩種技術的完整測試程式碼以及它們各自的錯誤簽名在下面的區塊中依次列出。對於我的範例，我使用了免費提供的 Noto Sans Devanagari（常規重量），可在此處取得：連結到 Noto Sans Devanagari 的 google fonts GitHub
技術1使用 Devanagari 和 HarfBuzz（如果使用 Node 渲染器編譯，則不會出現 lua 錯誤）：
\documentclass{article}
\usepackage[lmargin=0.5in,tmargin=0.5in,rmargin=0.5in,bmargin=0.5in]{geometry}
\usepackage{fontspec}
\usepackage{microtype}
%\newfontscript{Devanagari}{deva,dev2}
\newfontfamily{\devanagarifam}{Noto Sans Devanagari}[Script=Devanagari, Scale=1, Renderer=HarfBuzz]
\begin{document}
% Devanagari text is at the right end of following line
% of code, you might have to scroll right to read it
\setbox0=\hbox{Příliš žluťoučký \textit{kůň} úpěl \hbox{ďábelské} ódy difference diffierence. \devanagarifam एक गांव -- में मोहन नाम का लड़का रहता था। उसके पिताजी एक मामूली मजदूर थे।}
\directlua{
% local fontstyles = require "l4fontstyles"
local char = unicode.utf8.char
local glyph_id = node.id("glyph")
local glue_id = node.id("glue")
local hlist_id = node.id("hlist")
local vlist_id = node.id("vlist")
local disc_id = node.id("disc")
local minglue = tex.sp("0.2em")
local usedcharacters = {}
local identifiers = fonts.hashes.identifiers
local function get_unicode(xchar,font_id)
local current = {}
local uchar = identifiers[font_id].characters[xchar].tounicode
for i= 1, string.len(uchar), 4 do
local cchar = string.sub(uchar, i, i + 3)
print(xchar,uchar,cchar, font_id, i)
table.insert(current,char(tonumber(cchar,16)))
end
return current
end
local function nodeText(n)
local t = {}
for x in node.traverse(n) do
% glyph node
if x.id == glyph_id then
% local currentchar = fonts.hashes.identifiers[x.font].characters[x.char].tounicode
local chars = get_unicode(x.char,x.font)
for _, current_char in ipairs(chars) do
table.insert(t,current_char)
end
% glue node
elseif x.id == glue_id and node.getglue(x) > minglue then
table.insert(t," ")
% discretionaries
elseif x.id == disc_id then
table.insert(t, nodeText(x.replace))
% recursivelly process hlist and vlist nodes
elseif x.id == hlist_id or x.id == vlist_id then
table.insert(t,nodeText(x.head))
end
end
return table.concat(t)
end
local n = tex.getbox(0)
print(nodeText(n.head))
local f = io.open("hello.txt","w")
f:write(nodeText(n.head))
f:close()
}
\box0
\end{document}
Technique-1（HarfBuzz 渲染器）的錯誤簽名：
[\directlua]:1: bad argument #1 to 'len' (string expected, got nil)
stack traceback:
[C]: in function 'string.len'
[\directlua]:1: in upvalue 'get_unicode'
[\directlua]:1: in local 'nodeText'
[\directlua]:1: in main chunk.
l.62 }
技術2使用 Devanagari 和 HarfBuzz（如果使用 Node 渲染器編譯，則不會出現 lua 錯誤）：
\documentclass{article}
\usepackage[lmargin=0.5in,tmargin=0.5in,rmargin=0.5in,bmargin=0.5in]{geometry}
\usepackage{fontspec}
\usepackage{microtype}
%\newfontscript{Devanagari}{deva,dev2}
\newfontfamily{\devanagarifam}{Noto Sans Devanagari}[Script=Devanagari, Scale=1, Renderer=HarfBuzz]
\begin{document}
% Devanagari text is at the right end of following line
% of code, you might have to scroll right to read it
\setbox0=\hbox{Příliš žluťoučký \textit{kůň} úpěl \hbox{ďábelské} ódy difference diffierence. \devanagarifam एक गांव -- में मोहन नाम का लड़का रहता था। उसके पिताजी एक मामूली मजदूर थे।}
\directlua{
local glyph_id = node.id("glyph")
local disc_id = node.id("disc")
local glue_id = node.id("glue")
local hlist_id = node.id("hlist")
local vlist_id = node.id("vlist")
local minglue = tex.sp("0.2em")
local function nodeText(n)
local t = {}
for x in node.traverse(n) do
% glyph node
if x.id == glyph_id then
if bit32.band(x.subtype,2) \csstring~=0 and unicode.utf8.char(x.char) \csstring~="“" and unicode.utf8.char(x.char) \csstring~="”" then %
for g in node.traverse_id(glyph_id,x.components) do
if bit32.band(g.subtype, 2) \csstring~=0 then
for gc in node.traverse_id(glyph_id,g.components) do
table.insert(t,unicode.utf8.char(gc.char))
end
else
table.insert(t,unicode.utf8.char(g.char))
end
end
else
table.insert(t,unicode.utf8.char(x.char))
end
% disc node
elseif x.id == disc_id then
for g in node.traverse_id(glyph_id,x.replace) do
if bit32.band(g.subtype, 2) \csstring~=0 then
for gc in node.traverse_id(glyph_id,g.components) do
table.insert(t,unicode.utf8.char(gc.char))
end
else
table.insert(t,unicode.utf8.char(g.char))
end
end
% glue node
elseif x.id == glue_id and node.getglue(x) > minglue then
table.insert(t," ")
elseif x.id == hlist_id or x.id == vlist_id then
table.insert(t,nodeText(x.head))
end
end
return table.concat(t)
end
local n = tex.getbox(0)
print(nodeText(n.head))
local f = io.open("hello.txt","w")
f:write(nodeText(n.head))
f:close()
}
\box0
\end{document}
Technique-2（HarfBuzz 渲染器）的錯誤簽名：
[\directlua]:1: bad argument #1 to 'char' (invalid value)
stack traceback:
[C]: in field 'char'
[\directlua]:1: in local 'nodeText'
[\directlua]:1: in main chunk.
l.64 }
答案1
在節點模式下，恢復全文通常是不可能的，因為您會得到整形輸出，並且整形字形無法唯一地映射回輸入文字。您只能使用 tounicode 值來近似它。這些對應到實際的 PDF 檔案 ToUnicode CMap 條目，因此遵循其字形到 Unicode 映射的受限模型：每個字形相當於固定的 unicode 代碼點序列。這些映射按渲染順序串聯起來。正如您所看到的，此模型不足以將梵文字形映射到輸入文字。
您可以使用harfmode 來避免此問題：harfmode 不受此有限模型的影響，因為它不僅為您提供形狀的字形列表，而且還創建PDF 標記內容ActualText 條目，覆蓋序列中的ToUnicode 映射，而這些條目可能不正確透過 ToUnicode 建模。可以使用該屬性從 Lua 程式碼中查詢此映射所需的資料glyph_data。 （這是一個未記錄的實作細節，將來可能會改變）
如果 xou 想要從任何文字中提取盡可能多的內容，您可以在 Lua 程式碼中結合使用此基於屬性和基於 ToUnicode 的方法：
建立一個extracttext.lua文件
local type = type
local char = utf8.char
local unpack = table.unpack
local getproperty = node.getproperty
local getfont = font.getfont
local is_glyph = node.is_glyph
-- tounicode id UTF-16 in hex, so we need to handle surrogate pairs...
local utf16hex_to_utf8 do -- Untested, but should more or less work
local l = lpeg
local tonumber = tonumber
local hex = l.R('09', 'af', 'AF')
local byte = hex * hex
local simple = byte * byte / function(s) return char(tonumber(s, 16)) end
local surrogate = l.S'Dd' * l.C(l.R('89', 'AB', 'ab') * byte)
* l.S'Dd' * l.C(l.R('CF', 'cf') * byte) / function(high, low)
return char(0x10000 + ((tonumber(high, 16) & 0x3FF) << 10 | (tonumber(low, 16) & 0x3FF)))
end
utf16hex_to_utf8 = l.Cs((surrogate + simple)^0)
end
-- First the non-harf case
-- Standard caching setup
local identity_table = setmetatable({}, {__index = function(_, id) return char(id) end})
local cached_text = setmetatable({}, {__index = function(t, fid)
local fontdir = getfont(fid)
local characters = fontdir and fontdir.tounicode == 1 and fontdir.characters
local font_cache = characters and setmetatable({}, {__index = function(tt, slot)
local character = characters[slot]
local text = character and character.tounicode or slot
-- At this point we have the tounicode value in text. This can have different forms.
-- The order in the if ... elseif chain is based on how likely it is to encounter them.
-- This is a small performance optimization.
local t = type(text)
if t == 'string' then
text = utf16hex_to_utf8:match(text)
elseif t == 'number' then
text = char(text)
elseif t == 'table' then
text = char(unpack(text)) -- I haven't tested this case, but it should work
end
tt[slot] = text
return text
end}) or identity_table
t[fid] = font_cache
return font_cache
end})
-- Now the tounicode case just has to look up the value
local function from_tounicode(n)
local slot, fid = is_glyph(n)
return cached_text[fid][slot]
end
-- Now the traversing stuff. Nothing interesting to see here except for the
-- glyph case
local traverse = node.traverse
local glyph, glue, disc, hlist, vlist = node.id'glyph', node.id'glue', node.id'disc', node.id'hlist', node.id'vlist'
local extract_text_vlist
-- We could replace i by #t+1 but this should be slightly faster
local function real_extract_text(head, t, i)
for n, id in traverse(head) do
if id == glyph then
-- First handle harf mode: Look for a glyph_info property. If that does not exists
-- use from_tounicode. glyph_info will sometimes/often be an empty string. That's
-- intentional and it should *not* trigger a fallback. The actual mapping will be
-- contained in surrounding chars.
local props = getproperty(n)
t[i] = props and props.glyph_info or from_tounicode(n)
i = i + 1
elseif id == glue then
if n.width > 1001 then -- 1001 is arbitrary but sufficiently high to be bigger than most weird glue uses
t[i] = ' '
i = i + 1
end
elseif id == disc then
i = real_extract_text(n.replace, t, i)
elseif id == hlist then
i = real_extract_text(n.head, t, i)
elseif id == vlist then
i = extract_text_vlist(n.head, t, i)
end
end
return i
end
function extract_text_vlist(head, t, i) -- glue should not become a space here
for n, id in traverse(head) do
if id == hlist then
i = real_extract_text(n.head, t, i)
elseif id == vlist then
i = extract_text_vlist(n.head, t, i)
end
end
return i
end
return function(list)
local t = {}
real_extract_text(list.head, t, 1)
return table.concat(t)
end
這可以用作普通的 Lua 模組：
\documentclass{article}
\usepackage{fontspec}
\newfontfamily{\devharf}{Noto Sans Devanagari}[Script=Devanagari, Renderer=HarfBuzz]
\newfontfamily{\devnode}{Noto Sans Devanagari}[Script=Devanagari, Renderer=Node]
\begin{document}
% Devanagari text is at the right end of following line
% of code, you might have to scroll right to read it
\setbox0=\hbox{Příliš žluťoučký \textit{kůň} úpěl \hbox{ďábelské} ódy difference diffierence. \devharf एक गांव -- में मोहन नाम का लड़का रहता था। उसके पिताजी एक मामूली मजदूर थे।}
\setbox1=\hbox{Příliš žluťoučký \textit{kůň} úpěl \hbox{ďábelské} ódy difference diffierence. \devnode एक गांव -- में मोहन नाम का लड़का रहता था। उसके पिताजी एक मामूली मजदूर थे।}
\directlua{
local extracttext = require'extracttext'
local f = io.open("hello.harf.txt","w") % Can reproduce the full input text
f:write(extracttext(tex.getbox(0)))
f:close()
f = io.open("hello.node.txt","w") % In node mode, we only get an approximation
f:write(extracttext(tex.getbox(1)))
f:close()
}
\box0
\box1
\end{document}
更一般的注意事項：正如您所看到的，從形狀列表中獲取文本時涉及到一些工作，特別是在 ToUnicode 情況下，我們必須映射代理對和其他內容。這主要是因為形狀文字是不是旨在用於此類用途。一旦字形節點受到保護（又稱 subtype(n) >= 256 或not is_char(n)is true），.char條目將不再包含 Unicode 值，而是包含內部標識符，該.font條目可能不再是您期望的值，並且某些字形可能無法表示完全作為字形。在大多數情況下，您想要實際訪問框後面的文字而不僅僅是文字的視覺顯示，您確實想要攔截列表前它首先就成形了。
答案2
我不太了解 Luaotfload 如何處理 HarfBuzz 字體，但我能夠找到獲取字段的方法tounicode，這要歸功於table.serialize.所以我為 Harfbuzz 改編的原始程式碼如下所示：
\documentclass{article}
\usepackage[lmargin=0.5in,tmargin=0.5in,rmargin=0.5in,bmargin=0.5in]{geometry}
\usepackage{fontspec}
\usepackage{microtype}
\usepackage{luacode}
%\newfontscript{Devanagari}{deva,dev2}
\newfontfamily{\devanagarifam}{Noto Sans Devanagari}[Script=Devanagari, Scale=1, Renderer=HarfBuzz]
\newfontfamily{\arabicfam}{Amiri}[Script=Arabic, Scale=1, Renderer=HarfBuzz]
\begin{document}
\setbox0=\hbox{Příliš žluťoučký \textit{kůň} úpěl \hbox{ďábelské} ódy difference diffierence. \devanagarifam एक गांव -- में मोहन नाम का लड़का रहता था। उसके पिताजी एक मामूली मजदूर थे।}
\setbox1=\hbox{\arabicfam \textdir TRT هذه المقالة عن براغ. لتصفح عناوين مشابهة، انظر براغ (توضيح).}
\begin{luacode*}
-- local fontstyles = require "l4fontstyles"
local char = unicode.utf8.char
local glyph_id = node.id("glyph")
local glue_id = node.id("glue")
local hlist_id = node.id("hlist")
local vlist_id = node.id("vlist")
local disc_id = node.id("disc")
local minglue = tex.sp("0.2em")
local usedcharacters = {}
local identifiers = fonts.hashes.identifiers
local fontcache = {}
local function to_unicode_chars(uchar)
local uchar = uchar or ""
-- put characters into a table
local current = {}
-- each codepoint is 4 bytes long, we loop over tounicode entry and cut it into 4 bytes chunks
for i= 1, string.len(uchar), 4 do
local cchar = string.sub(uchar, i, i + 3)
-- codepoint is hex string, we need to convert it to number ad then to UTF8 char
table.insert(current,char(tonumber(cchar,16)))
end
return current
end
-- cache character lookup, to speed up things
local function get_character_from_cache(xchar, font_id)
local current_font = fontcache[font_id] or {characters = {}}
fontcache[font_id] = current_font -- initialize font cache for the current font if it doesn't exist
return current_font.characters[xchar]
end
-- save characters to cache for faster lookup
local function save_character_to_cache(xchar, font_id, replace)
fontcache[font_id][xchar] = replace
-- return value
return replace
end
local function initialize_harfbuzz_cache(font_id, hb)
-- save some harfbuzz tables for faster lookup
local current_font = fontcache[font_id]
-- the unicode data can be in two places
-- 1. hb.shared.glyphs[glyphid].backmap
current_font.glyphs = current_font.glyphs or hb.shared.glyphs
-- 2. hb.shared.unicodes
-- it contains mapping between Unicode and glyph id
-- we must create new table that contains reverse mapping
if not current_font.backmap then
current_font.backmap = {}
for k,v in pairs(hb.shared.unicodes) do
current_font.backmap[v] = k
end
end
-- save it back to the font cache
fontcache[font_id] = current_font
return current_font.glyphs, current_font.backmap
end
local function get_unicode(xchar,font_id)
-- try to load character from cache first
local current_char = get_character_from_cache(xchar, font_id)
if current_char then return current_char end
-- get tounicode for non HarfBuzz fonts
local characters = identifiers[font_id].characters
local uchar = characters[xchar].tounicode
-- stop processing if tounicode exists
if uchar then return save_character_to_cache(xchar, font_id, to_unicode_chars(uchar)) end
-- detect if font is processed by Harfbuzz
local hb = identifiers[font_id].hb
-- try HarfBuzz data
if not uchar and hb then
-- get glyph index of the character
local index = characters[xchar].index
-- load HarfBuzz tables from cache
local glyphs, backmap = initialize_harfbuzz_cache(font_id, hb)
-- get tounicode field from HarfBuzz glyph info
local tounicode = glyphs[index].tounicode
if tounicode then
return save_character_to_cache(xchar, font_id, to_unicode_chars(tounicode))
end
-- if this fails, try backmap, which contains mapping between glyph index and Unicode
local backuni = backmap[index]
if backuni then
return save_character_to_cache(xchar, font_id, {char(backuni)})
end
-- if this fails too, discard this character
return save_character_to_cache(xchar, font_id, {})
end
-- return just the original char if everything else fails
return save_character_to_cache(xchar, font_id, {char(xchar)})
end
local function nodeText(n)
-- output buffer
local t = {}
for x in node.traverse(n) do
-- glyph node
if x.id == glyph_id then
-- get table with characters for current node.char
local chars = get_unicode(x.char,x.font)
for _, current_char in ipairs(chars) do
-- save characters to the output buffer
table.insert(t,current_char)
end
-- glue node
elseif x.id == glue_id and node.getglue(x) > minglue then
table.insert(t," ")
-- discretionaries
elseif x.id == disc_id then
table.insert(t, nodeText(x.replace))
-- recursivelly process hlist and vlist nodes
elseif x.id == hlist_id or x.id == vlist_id then
table.insert(t,nodeText(x.head))
end
end
return table.concat(t)
end
local n = tex.getbox(0)
local n1 = tex.getbox(1)
print(nodeText(n.head))
local f = io.open("hello.txt","w")
f:write(nodeText(n.head))
f:write(nodeText(n1.head))
f:close()
\end{luacode*}
\box0
\box1
\end{document}
我還添加了來自的阿拉伯語樣本維基百科。以下是 的內容hello.txt：
Příliš žluťoučký kůň úpěl ďábelské ódy difference diffierence. एक गांव -- में मोहन नाम का लड़का रहता था। उसके पताजी एक मामूली मजदूर थे।هذه المقالة عن براغ. لتصفح عناوين مشابهة، انظر براغ (توضيح).
兩個重要的功能是
local function to_unicode_chars(uchar)
local uchar = uchar or ""
local current = {}
for i= 1, string.len(uchar), 4 do
local cchar = string.sub(uchar, i, i + 3)
table.insert(current,char(tonumber(cchar,16)))
end
return current
end
to_unicode_chars函數將 to_unicode 條目拆分為四個位元組區塊，然後將其轉換為 UTF 8 字元。它還可以處理沒有tounicode條目的字形，在這種情況下它只會傳回空字串。
local function get_unicode(xchar,font_id)
-- try to load character from cache first
local current_char = get_character_from_cache(xchar, font_id)
if current_char then return current_char end
-- get tounicode for non HarfBuzz fonts
local characters = identifiers[font_id].characters
local uchar = characters[xchar].tounicode
-- stop processing if tounicode exists
if uchar then return save_character_to_cache(xchar, font_id, to_unicode_chars(uchar)) end
-- detect if font is processed by Harfbuzz
local hb = identifiers[font_id].hb
-- try HarfBuzz data
if not uchar and hb then
-- get glyph index of the character
local index = characters[xchar].index
-- load HarfBuzz tables from cache
local glyphs, backmap = initialize_harfbuzz_cache(font_id, hb)
-- get tounicode field from HarfBuzz glyph info
local tounicode = glyphs[index].tounicode
if tounicode then
return save_character_to_cache(xchar, font_id, to_unicode_chars(tounicode))
end
-- if this fails, try backmap, which contains mapping between glyph index and Unicode
local backuni = backmap[index]
if backuni then
return save_character_to_cache(xchar, font_id, {char(backuni)})
end
-- if this fails too, discard this character
return save_character_to_cache(xchar, font_id, {})
end
-- return just the original char if everything else fails
return save_character_to_cache(xchar, font_id, {char(xchar)})
end
該函數首先嘗試從目前字體資訊載入 Uniocode 資料。如果失敗，它會嘗試在 Harfbuzz 表中尋找。大多數字元tounicode在表中都有映射glyphs。如果不可用，它會嘗試該unicodes表，其中包含字形索引和 Unicode 之間的對應。如果連這個都失敗了，那我們就丟棄這個角色。