ELKstack基础入门
ELK简介
通俗来讲，ELK是由Elasticsearch、Logstash、Kibana 三个开源软件的组成的一个组合体，这三个软件当中，每个软件用于完成不同的功能，ELK 又称为ELK stack，官方域名为elastic.co，ELK stack的主要优点有如下几个：
- 1.处理方式灵活： elasticsearch是实时全文索引，具有强大的搜索功能
- 2.配置相对简单：elasticsearch全部使用JSON 接口，logstash使用模块配置，kibana的配置文件部分更简单。
- 3.检索性能高效：基于优秀的设计，虽然每次查询都是实时，但是也可以达到百亿级数据的查询秒级响应。
- 4.集群线性扩展：elasticsearch和logstash都可以灵活线性扩展
- 5.前端操作绚丽：kibana的前端设计比较绚丽，而且操作简单
日志收集软件
- ELKstack
- flume
- 日志易
什么是elasticsearch
存储日志
数据库：是一个高度可扩展的开源全文搜索和分析引擎，它可实现数据的实时全文搜索搜索、支持分布式可实现高可用、提供API接口，可以处理大规模日志数据，比如Nginx、Tomcat、系统日志等功能。
什么是logstash（filebeat）
收集日志，转发日志
可以通过插件实现日志收集和转发，支持日志过滤，支持普通log、自定义json格式的日志解析。
什么是Kibana
主要是通过接口调用elasticsearch的数据，并进行前端数据可视化的展现。
部署Elasticsearch
环境准备
主机名
外网IP
内网IP
角色
应用
ELKstack01
10.0.0.81
172.16.1.81
ES日志存储数据库
JDK、elasticsearch
ELKstack02
10.0.0.82
172.16.1.82
ES日志存储数据库
JDK、elasticsearch
| 主机名 | 外网IP | 内网IP | 角色 | 应用 |
|---|---|---|---|---|
| ELKstack01 | 10.0.0.81 | 172.16.1.81 | ES日志存储数据库 | JDK、elasticsearch |
| ELKstack02 | 10.0.0.82 | 172.16.1.82 | ES日志存储数据库 | JDK、elasticsearch |
##下载JDK包
[root@elk1 ~]# wget https://download.oracle.com/java/20/latest/jdk-20_linux-x64_bin.tar.gz
## 解压JDK包
[root@elk1 ~]# tar xf jdk-20_linux-x64_bin.tar.gz
## 做软连接
[root@elk1 ~]# rm -fr /usr/bin/java
[root@elk1 ~]# ln -s /root/jdk-20.0.1/bin/java /usr/bin/
## 检查是否加载成功
[root@elk1 ~]# java -version
java version "20.0.1" 2023-04-18
Java(TM) SE Runtime Environment (build 20.0.1+9-29)
Java HotSpot(TM) 64-Bit Server VM (build 20.0.1+9-29, mixed mode, sharing)
## 偷懒装法
[root@elk1 ~]# yum install -y java
[root@elk2 ~]# yum install -y java
## 更换ES官方源
[root@elk1 ~]# vim /etc/yum.repos.d/es.repo
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
#拷贝到10.0.0.82
[root@elk1 ~]# scp /etc/yum.repos.d/es.repo 10.0.0.82:/etc/yum.repos.d/
The authenticity of host '10.0.0.82 (10.0.0.82)' can't be established.
ECDSA key fingerprint is SHA256:5KRH49USplFqTLegk1cdAiaJ7C1EJR+PjWAswdkRE8E.
ECDSA key fingerprint is MD5:b3:4f:97:67:42:72:0d:12:d2:69:3c:86:9e:2c:fd:fa.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.82' (ECDSA) to the list of known hosts.
root@10.0.0.82's password:
es.repo
# 更新缓存
[root@elk1 ~]# yum makecache
[root@elk2 ~]# yum makecache
# 下载es
[root@elk1 ~]# yum install -y elasticsearch
[root@elk2 ~]# yum install -y elasticsearch
# 配置es的配置文件
[root@elk1 ~]# vim /etc/elasticsearch/elasticsearch.yml
17 cluster.name: elkstack
23 node.name: es01
33 path.data: /data/es/data
37 path.logs: /data/es/logs
43 #bootstrap.memory_lock: true 如果服务起不来就改false
55 network.host: 0.0.0.0
59 http.port: 9200
68 discovery.zen.ping.unicast.hosts: ["10.0.0.81", "10.0.0.82"]
[root@elk2 ~]# vim /etc/elasticsearch/elasticsearch.yml
17 cluster.name: elkstack
23 node.name: es02
33 path.data: /data/es/data
37 path.logs: /data/es/logs
43 #bootstrap.memory_lock: true 如果服务起不来就改false
55 network.host: 0.0.0.0
59 http.port: 9200
68 discovery.zen.ping.unicast.hosts: ["10.0.0.81", "10.0.0.82"]
# 创建日志和数据存放目录
[root@elk1 ~]# mkdir -p /data/es/data
[root@elk1 ~]# mkdir -p /data/es/logs
[root@elk2 ~]# mkdir -p /data/es/data
[root@elk2 ~]# mkdir -p /data/es/logs
# 修改启动脚本
[root@elk1 ~]# vim /usr/lib/systemd/system/elasticsearch.service
[root@elk2 ~]# vim /usr/lib/systemd/system/elasticsearch.service
LimitMEMLOCK=infinity # 修改内存限制（去掉此行注释，没有就忽略）
# 授权
[root@elk1 ~]# chown -R elasticsearch.elasticsearch /data/
[root@elk2 ~]# chown -R elasticsearch.elasticsearch /data/
# 优化文件描述符
[root@elk1 ~]# vim /etc/security/limits.conf
[root@elk2 ~]# vim /etc/security/limits.conf
最后一行加入
* soft memlock unlimited
* hard memlock unlimited
* soft nofile 131072
* hard nofile 131072
# 设置JVM最大最小内存限制
[root@elk1 ~]# vim /etc/elasticsearch/jvm.options
[root@elk2 ~]# vim /etc/elasticsearch/jvm.options
把-Xms2g改成-Xms1g
把-Xmx2g改成-Xmx1g
# 启动es
[root@elk1 ~]# systemctl start elasticsearch.service
[root@elk2 ~]# systemctl start elasticsearch.service
安装elasticsearch插件
插件是为了完成不同的功能，官方提供了一些插件但大部分是收费的，另外也有一些开发爱好者提供的
插件，可以实现对elasticsearch集群的状态监控与管理配置等功能，我们现在要安装的是Elasticsearch
的head插件，此插件提供elasticsearch的web界面功能。
安装Elasticsearch的head插件时，要安装npm，npm的全称是Node Package Manager，是随同
NodeJS一起安装的包管理和分发工具，它很方便让JavaScript开发者下载、安装、上传以及管理已经安
装的包。
在Elasticsearch 5.x版本以后不再支持直接安装head插件，而是需要通过启动一个服务方式。
# 安装npm
[root@elk2 ~]# yum install -y npm
npm 前端语法 //node
head插件多是一个前端的代码
python-pip
# 安装git命令
[root@elk2 ~]# yum install -y git
## 下载head插件
[root@elk2 ~]# git clone https://github.com/mobz/elasticsearch-head.git
Cloning into 'elasticsearch-head'...
remote: Enumerating objects: 4377, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 4377 (delta 12), reused 34 (delta 12), pack-reused 4337
Receiving objects: 100% (4377/4377), 2.54 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2429/2429), done.
## 下载nodejs
[root@elk2 ~]# wget https://nodejs.org/dist/v16.13.0/node-v16.13.0-linux-x64.tar.xz
--2023-07-10 21:15:14-- https://nodejs.org/dist/v16.13.0/node-v16.13.0-linux-x64.tar.xz
Resolving nodejs.org (nodejs.org)... 104.20.23.46, 104.20.22.46, 2606:4700:10::6814:172e, ...
Connecting to nodejs.org (nodejs.org)|104.20.23.46|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21605928 (21M) [application/x-xz]
Saving to: ‘node-v16.13.0-linux-x64.tar.xz’
100%[===================================================================>] 21,605,928 8.06MB/s in 2.6s
2023-07-10 21:15:17 (8.06 MB/s) - ‘node-v16.13.0-linux-x64.tar.xz’ saved [21605928/21605928]
## 解压nodejs
[root@elk2 ~]# tar xf node-v16.13.0-linux-x64.tar.xz
[root@elk2 ~]# ll
total 21112
-rw-------. 1 root root 1512 Apr 11 16:23 anaconda-ks.cfg
drwxr-xr-x 8 root root 4096 Jul 10 21:14 elasticsearch-head
-rw-r--r-- 1 root root 195 Apr 12 09:11 ip.sh
drwxr-xr-x 6 1001 1001 108 Oct 26 2021 node-v16.13.0-linux-x64
-rw-r--r-- 1 root root 21605928 Oct 26 2021 node-v16.13.0-linux-x64.tar.xz
## 创建app目录吧nodejs移到app目录下
[root@elk2 ~]# mkdir /app
[root@elk2 ~]# mv node-v16.13.0-linux-x64 /app/
## 做软连接
[root@elk2 app]# ln -s /app/node-v16.13.0-linux-x64/ /app/node
[root@elk2 app]# ll
total 0
lrwxrwxrwx 1 root root 29 Jul 10 21:18 node -> /app/node-v16.13.0-linux-x64/
drwxr-xr-x 6 1001 1001 108 Oct 26 2021 node-v16.13.0-linux-x64
# 添加环境变量
[root@elk2 app]# vim /etc/profile.d/node.sh
expr PATH="/app/node/bin:$PATH"
# 生效配置文件
[root@elk2 app]# source /etc/profile
PATH=/app/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
# 查看版本
[root@elk2 app]# npm --version
8.1.0
# 进入head插件目录
[root@elk2 app]# cd
[root@elk2 ~]# cd elasticsearch-head/
# 换源
[root@elk2 elasticsearch-head]# npm config set registry=https://registry.npm.taobao.org
# 查看当前包下的镜像源
[root@elk2 elasticsearch-head]# npm config get registry
https://registry.npm.taobao.org/
# 安装换源命令
[root@elk2 elasticsearch-head]# npm install -g nrm open@8.4.2 --save
added 17 packages in 2s
npm notice
npm notice New major version of npm available! 8.19.2 -> 9.8.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v9.8.0
npm notice Run npm install -g npm@9.8.0 to update!
npm notice
# 查看可用源
[root@elk2 elasticsearch-head]# nrm ls
npm ---------- https://registry.npmjs.org/
yarn --------- https://registry.yarnpkg.com/
tencent ------ https://mirrors.cloud.tencent.com/npm/
cnpm --------- https://r.cnpmjs.org/
taobao ------- https://registry.npmmirror.com/
npmMirror ---- https://skimdb.npmjs.com/registry/
# 切换到taobao源
[root@elk2 elasticsearch-head]# nrm use taobao
SUCCESS The registry has been changed to 'taobao'.
## 安装
[root@elk2 elasticsearch-head]# npm install grunt -save -no-fund
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE package: 'karma@1.3.0',
npm WARN EBADENGINE required: { node: '0.10 || 0.12 || 4 || 5 || 6' },
npm WARN EBADENGINE current: { node: 'v16.18.1', npm: '8.19.2' }
npm WARN EBADENGINE }
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE package: 'http2@3.3.7',
npm WARN EBADENGINE required: { node: '>=0.12.0 <9.0.0' },
npm WARN EBADENGINE current: { node: 'v16.18.1', npm: '8.19.2' }
npm WARN EBADENGINE }
npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated chokidar@1.7.0: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated phantomjs-prebuilt@2.1.16: this package is now deprecated
npm WARN deprecated http2@3.3.7: Use the built-in module in node 9.0.0 or newer, instead
npm WARN deprecated json3@3.2.6: Please use the native JSON object instead of JSON 3
npm WARN deprecated coffee-script@1.10.0: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN deprecated log4js@0.6.38: 0.x is no longer supported. Please upgrade to 6.x or higher.
npm WARN deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
npm ERR! code 1
npm ERR! path /root/elasticsearch-head/node_modules/phantomjs-prebuilt
npm ERR! command failed
npm ERR! command sh -c -- node install.js
npm ERR! PhantomJS not found on PATH
npm ERR! Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
npm ERR! Saving to /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
npm ERR! Receiving...
npm ERR!
npm ERR! Received 22866K total.
npm ERR! Extracting tar contents (via spawned process)
npm ERR! Error extracting archive
npm ERR! Phantom installation failed Error: Command failed: tar jxf /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
npm ERR! tar (child): bzip2: Cannot exec: No such file or directory
npm ERR! tar (child): Error is not recoverable: exiting now
npm ERR! tar: Child returned status 2
npm ERR! tar: Error is not recoverable: exiting now
npm ERR!
npm ERR! at ChildProcess.exithandler (node:child_process:402:12)
npm ERR! at ChildProcess.emit (node:events:513:28)
npm ERR! at maybeClose (node:internal/child_process:1100:16)
npm ERR! at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5) {
npm ERR! code: 2,
npm ERR! killed: false,
npm ERR! signal: null,
npm ERR! cmd: 'tar jxf /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2'
npm ERR! } Error: Command failed: tar jxf /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
npm ERR! tar (child): bzip2: Cannot exec: No such file or directory
npm ERR! tar (child): Error is not recoverable: exiting now
npm ERR! tar: Child returned status 2
npm ERR! tar: Error is not recoverable: exiting now
npm ERR!
npm ERR! at ChildProcess.exithandler (node:child_process:402:12)
npm ERR! at ChildProcess.emit (node:events:513:28)
npm ERR! at maybeClose (node:internal/child_process:1100:16)
npm ERR! at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2023-07-10T13_26_17_737Z-debug-0.log
## 报错解决方法
[root@elk2 elasticsearch-head]# yum install -y bzip2
## 在执行一次
[root@elk2 elasticsearch-head]# npm install grunt -save -no-fund
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE package: 'karma@1.3.0',
npm WARN EBADENGINE required: { node: '0.10 || 0.12 || 4 || 5 || 6' },
npm WARN EBADENGINE current: { node: 'v16.18.1', npm: '8.19.2' }
npm WARN EBADENGINE }
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE package: 'http2@3.3.7',
npm WARN EBADENGINE required: { node: '>=0.12.0 <9.0.0' },
npm WARN EBADENGINE current: { node: 'v16.18.1', npm: '8.19.2' }
npm WARN EBADENGINE }
npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3
npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated chokidar@1.7.0: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated phantomjs-prebuilt@2.1.16: this package is now deprecated
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated http2@3.3.7: Use the built-in module in node 9.0.0 or newer, instead
npm WARN deprecated json3@3.2.6: Please use the native JSON object instead of JSON 3
npm WARN deprecated coffee-script@1.10.0: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN deprecated log4js@0.6.38: 0.x is no longer supported. Please upgrade to 6.x or higher.
npm WARN deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
added 540 packages in 20s
## 启动插件
[root@elk2 elasticsearch-head]# npm run start &
[1] 2102
[root@elk2 elasticsearch-head]#
> elasticsearch-head@0.0.0 start
> grunt server
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100
# 检查插件
[root@elk2 elasticsearch-head]# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9100 0.0.0.0:* LISTEN 2113/grunt
# 访问浏览器
10.0.0.82:9100!
# 这时还连接不上集群，要给es做跨域访问配置
[root@elk2 elasticsearch-head]# vim /etc/elasticsearch/elasticsearch.yml
末行加入
http.cors.enabled: true
http.cors.allow-origin: "*"
## 重启es
[root@elk2 elasticsearch-head]# systemctl restart elasticsearch.service
##重新访问10.0.0.82:9100
