logstash部署

Logstash是一个开源的数据收集引擎，可以水平伸缩，而且logstash整个ELK当中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指定的且可以是多个不同目的地。

# 安装jdk
[root@logstash ~]# yum install -y java
# logstash的下载
[root@logstash ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.3.0.rpm
# 安装logstash
[root@logstash ~]# yum localinstall logstash-5.3.0.rpm -y
# 授权
[root@logstash ~]# chown logstash.logstash /usr/share/logstash/

logstash的插件

  • INPUT 输入插件 日志来自哪里
  • OUPUT 输出插件
INPUT支持事件源 OUTPUT支持输出源 CODEC编解码器支持编码
azure_event_hubs(微软云事件中心) elasticsearch(搜索引擎数据库) avro(数据序列化)
beats(filebeat日志收集工具) email(邮件) CEF(嵌入式框架)
elasticsearch(搜索引擎数据库) file(文件) es_bulk(ES中的bulk api)
file(文件) http(超文本传输协议) Json(数据序列化、格式化)
generator(生成器) kafka(基于java的消息队列) Json_lines(便于存储结构化)
heartbeat(高可用软件) rabbitmq(消息队列 OpenStack) line(行)
http_poller(http api) redis(缓存、消息队列、NoSQL) multiline(多行匹配)
jdbc(java连接数据库的驱动) s3*(存储) plain(纯文本，事件间无间隔)
kafka(基于Java的消息队列) sidout(标准输出) rubydebug(ruby语法格式)
rabbitmq(消息队列 OpenStack) tcp(传输控制协议)
redis(缓存、消息队列、NoSQL) udp(用户数据报协议)
s3*(存储)
stdin(标准输入)
syslog(系统日志)
tcp(传输控制协议)
udp(用户数据报协议)

测试标准输入和输出

标准输入 在终端打的字
标准输出 终端显示的字

[root@logstash ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}'
nginx        # 标准输入
{             
    "@timestamp" => 2023-07-12T10:04:47.252Z,
      "@version" => "1",
          "host" => "logstash",
       "message" => "nginx"
}            # 标准输出

测试logstash标准输入到文件

[root@logstash ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { file { path => "/tmp/test_%{+YYYY.MM.dd}.log"}}'
nginx    #标准输入
php
[root@logstash ~]# cat /tmp/test_2023.07.12.log  # 标准输出到文件
{"@timestamp":"2023-07-12T10:08:29.345Z","@version":"1","host":"logstash","message":"nginx"}
{"@timestamp":"2023-07-12T10:08:33.782Z","@version":"1","host":"logstash","message":"php"}

测试标准输出到ES

[root@logstash ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{hosts => ["10.0.0.81:9200"] index => "ceshi_%{+YYYY.MM.dd}" }}'

18:13:43.722 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9603}
111

img

img

logstash收集系统日志到es

[root@logstash ~]# /usr/share/logstash/bin/logstash -e 'input { file { type => "www" path => "/var/log/messages" start_position => "beginning" } } output { elasticsearch { hosts => ["10.0.0.82:9200"] index => "www_%{+YYYY.MM.dd}" } }'

img

img

logstash收集nginx日志到es

# 添加输入命令执行
cat >>/etc/profile.d/logstash.sh<< EOF
export PATH='/usr/share/logstash/bin:$PATH'
EOF
source /etc/profile

logstash -e 'input { file { type => "nginx" path => "/var/log/nginx/access.log" start_position => "beginning" } } output { elasticsearch { hosts => ["10.0.0.81:9200"] index => "nginx_%{+YYYY.MM.dd}" } }'

img

logstash收集系统日志

系统日志输出至ES

[root@logstash ~]# cd /etc/logstash/conf.d/
[root@logstash conf.d]# vim message_file_es.conf
input{
    file{
        type => "msg_log"
        path => "/var/log/messages"
        start_position => "beginning"
    }
}
output{
    elasticsearch{
        hosts => ["10.0.0.82:9200"]
        index => "msg_log-2023-07-12"
    }
}
# 启动logstash
[root@logstash conf.d]# logstash -f /etc/logstash/conf.d/message_file_es.conf

img

# 检测配置文件语法
[root@logstash conf.d]# logstash -f /etc/logstash/conf.d/message_file_es.conf -t
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
Configuration OK
16:43:25.160 [LogStash::Runner] INFO  logstash.runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

# 放后台运行
logstash -f /etc/logstash/conf.d/message_file_es.conf &

logstash收集多个日志到es

[root@logstash conf.d]# vim message_file_es.conf
input{
    file{
        type => "msg"
        path => "/var/log/messages"
        start_position => "beginning"
    }
    file{
        type => "sec"
        path => "/var/log/secure"
        start_position => "beginning"
    }
}
output{
    elasticsearch{
        hosts => ["10.0.0.82:9200"]
        index => "msg-2023-07-12"
    }
    elasticsearch{
        hosts => ["10.0.0.82:9200"]
        index => "sec-2023-07-12"
    }
}

[root@logstash conf.d]# logstash -f /etc/logstash/conf.d/message_file_es.conf

img

优化日期

input{
    file{
        type => "msg_log"
        path => "/var/log/messages"
        start_position => "beginning"
    }
    file{
        type => "sec_log"
        path => "/var/log/secure"
        start_position => "beginning"
    }
}
output{
    elasticsearch{
        hosts => ["10.0.0.82:9200"]
        index => "msg_log-%{yyyy.MM.dd}"
    }
    elasticsearch{
        hosts => ["10.0.0.82:9200"]
        index => "sec_log-%{yyyy.MM.dd}"
    }
}

[root@logstash conf.d]# logstash -f /etc/logstash/conf.d/message_file_es.conf

img

优化index名字

input{
    file{
        type => "msg"
        path => "/var/log/messages"
        start_position => "beginning"
    }
    file{
        type => "sec"
        path => "/var/log/secure"
        start_position => "beginning"
    }
}
output{
    elasticsearch{
        hosts => ["10.0.0.82:9200"]
        index => "%{type}-%{yyyy.MM.dd}"
    }
}