☰

如何將多個位置的rsyslog日誌轉發到ELK並使其顯示在kibana中？

tag-icon • tag-icon centos rsyslog kibana
如何將多個位置的rsyslog日誌轉發到ELK並使其顯示在kibana中？

多個遠端主機的 rsyslog 伺服器範本注意事項 ---> 連結到之前回答的問題

@meuh，我發現這篇文章非常有用，因為我目前正在研究這個配置。

我已經完成了上面提到的步驟並且工作正常。

我現在有一個 ELK 設置，其中 rsyslog 將日誌轉發給它。

我的模板是：

$template
templmesg,"/data01/RemoteLogs/DLF/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"

$template mylogsec,"/data01/RemoteLogs/Logserver/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"

if $fromhost startswith "10.100.10" then ?templmesg
 & stop
 if $fromhost startswith "10.100.112" then ?mylogsec
 & stop

所以我有兩個儲存日誌的位置。

由於 DLF 和 Logserver 等日誌儲存的多個位置。 (ELK) 的 Kibana 不顯示從 rsyslog 接收的日誌。它僅從 DLF/dir 中的一個位置讀取日誌，而不是從 Logserver 中讀取。

現在我陷入困境，不知道如何將rsyslog日誌從多個位置轉發到ELK並使其在kibana中顯示；或者，rsyslog 中是否有我需要解決的特定配置？

下面是rsyslog設定檔：

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

$template templmesg,"/data01/RemoteLogs/DLF/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"
$template mylogsec,"/data01/RemoteLogs/DLF/Logserver/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"

#if $fromhost startswith "10.100.10" then ?templmesg 
#& stop
if $fromhost startswith "10.100.112" then ?mylogsec 
& stop

local0.*                                                        ?templmesg
local1.*                                                        ?templmesg
local2.*                                                        ?templmesg
local3.*                                                        ?templmesg
local4.*                                                        ?templmesg
local5.*                                                        ?templmesg
local6.*                                                        ?templmesg


template(name="json-template"
  type="list") {
   constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}


# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#
#$createDirs on

*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none;local2.none;local3.none;local4.none;local5.none;local6.none              ?templmesg

# The authpriv file has restricted access.
authpriv.*                                              ?templmesg

# Log all the mail messages in one place.
mail.*                                                  ?templmesg


# Log cron stuff
cron.*                                                  ?templmesg

# Everybody gets emergency messages
#*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                ?templmesg


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @10.100.10.30:10514;json-template
# ### end of the forwarding rule ###

答案1

由於rsyslog.conf設定檔是從上到下解析的，因此按照檔案中定義的順序對每個訊息按順序執行操作。在您的情況下發生的情況是，處理與測試相符的訊息$fromhost startswith "10.100.112"（即寫入「mylogsec」範本指定的日誌檔案），然後由語句丟棄stop。

這個問題的解決方案很簡單。在 rsyslog 刪除訊息之前，您必須將其轉送到遠端 Logstash 伺服器。您可以修改您的過濾器，如下所示：

if $fromhost startswith "10.100.112" then ?mylogsec 
& @10.100.10.30:10514;json-template
& stop

由於您在此處使用 JSON 範本進行轉發，因此您還需要將該範本的定義移至篩選器表達式之前。所以最終的結構如下：

$template templmesg...
$template mylogsec...

template(name="json-template"...

if $fromhost startswith "10.100.112" then ?mylogsec 
& @10.100.10.30:10514;json-template
& stop

完成後，重新啟動rsyslog守護程序以使變更生效。

相關內容