Logstash - 过滤器

Logstash过滤器 - 从简单和简单的步骤学习Logstash，从基本到高级概念，包括简介，ELK堆栈，安装，内部架构，收集日志，支持的输入，解析日志，过滤器，转换日志，输出阶段，支持的输出，插件，监控API，安全和监控。

Logstash在输入和输出之间的管道中间使用过滤器. Logstash度量的过滤器操纵和创建 Apache-Access 等事件.许多过滤插件用于管理Logstash中的事件.这里，在 Logstash聚合过滤器的示例中，我们过滤数据库中每个SQL事务的持续时间并计算总时间.

安装聚合过滤器插件

使用Logstash-plugin实用程序安装聚合过滤器插件. Logstash-plugin是Logstash中 bin文件夹中Windows的批处理文件.

 > logstash-plugin install logstash-filter-aggregate

logstash.conf

在此配置中，您可以看到三个'if'语句初始化，递增，和生成交易的总持续时间，即 sql_duration .聚合插件用于添加sql_duration，它存在于输入日志的每个事件中.

input {   file {      path => "C:/tpwork/logstash/bin/log/input.log"   }} filter {   grok {      match => [         "message", "%{LOGLEVEL:loglevel} -             %{NOTSPACE:taskid} - %{NOTSPACE:logger} -             %{WORD:label}( - %{INT:duration:int})?"       ]   }   if [logger] == "TRANSACTION_START" {      aggregate {         task_id => "%{taskid}"         code => "map['sql_duration'] = 0"         map_action => "create"      }   }   if [logger] == "SQL" {      aggregate {         task_id => "%{taskid}"         code => "map['sql_duration'] ||= 0 ;            map['sql_duration'] += event.get('duration')"      }   }   if [logger] == "TRANSACTION_END" {      aggregate {         task_id => "%{taskid}"         code => "event.set('sql_duration', map['sql_duration'])"         end_of_task => true         timeout => 120      }   }}output {   file {      path => "C:/tpwork/logstash/bin/log/output.log"       }}

运行Logstash

我们可以使用以下命令运行Logstash.

> logstash -f logstash.conf

input.log

以下代码块显示输入日志数据.

INFO - 48566 - TRANSACTION_START - startINFO - 48566 - SQL - transaction1 - 320INFO - 48566 - SQL - transaction1 - 200INFO - 48566 - TRANSACTION_END - end

output.log

如配置文件中所指定的，最后一个'if'语句所在的记录器是 - TRANSACTION_END，它打印总事务时间或sql_duration.这在output.log中以黄色突出显示.

{   "path":"C:/tpwork/logstash/bin/log/input.log","@timestamp": "2016-12-22T19:04:37.214Z",   "loglevel":"INFO","logger":"TRANSACTION_START","@version": "1","host":"wcnlab-PC",   "message":"8566 - TRANSACTION_START - start\r","tags":[]}{   "duration":320,"path":"C:/tpwork/logstash/bin/log/input.log",   "@timestamp":"2016-12-22T19:04:38.366Z","loglevel":"INFO","logger":"SQL",   "@version":"1","host":"wcnlab-PC","label":"transaction1",   "message":" INFO - 48566 - SQL - transaction1 - 320\r","taskid":"48566","tags":[]}{   "duration":200,"path":"C:/tpwork/logstash/bin/log/input.log",   "@timestamp":"2016-12-22T19:04:38.373Z","loglevel":"INFO","logger":"SQL",   "@version":"1","host":"wcnlab-PC","label":"transaction1",   "message":" INFO - 48566 - SQL - transaction1 - 200\r","taskid":"48566","tags":[]}{   "sql_duration":520,"path":"C:/tpwork/logstash/bin/log/input.log",   "@timestamp":"2016-12-22T19:04:38.380Z","loglevel":"INFO","logger":"TRANSACTION_END",   "@version":"1","host":"wcnlab-PC","label":"end",   "message":" INFO - 48566 - TRANSACTION_END - end\r","taskid":"48566","tags":[]}