- 环境准备
- 配置 elk01 服务器
- 配置 elasticsearch
- 配置 elk02 服务器
- 配置 elasticsearch
- 配置 lb01 服务器
- 配置 nginx
- 配置 keepalived
- 配置 java 环境
- 配置 logstash
- 配置 lb02 服务器
- 配置 nginx
- 配置 keepalived
- 配置 kibana
- 配置 wbe01 服务器
- 配置 nginx
- 配置 filebeat
- 配置 java 环境
- 配置 zookeeper
- 配置 kafka
- 配置 wbe02 服务器
- 配置 nginx
- 配置 filebeat
- 配置 java 环境
- 配置 zookeeper
- 配置 kafka
- kafka 检查内容展示
- kibana
环境准备
主机名
外网 IP
内网 IP
应用
内存
lb01
10.0.0.5
172.16.1.5
nginx、logstash7.9
2G
lb02
10.0.0.6
172.16.1.6
nginx、kibana7.9
2G
web01
10.0.0.7
172.16.1.7
nginx、filebeat7.9、kafka2.12
2G
web02
10.0.0.8
172.16.1.8
nginx、filebeat7.9、kafka2.12
2G
elk01
10.0.0.81
172.16.1.81
elasticsearch7.9
2G
elk02
10.0.0.82
172.16.1.82
elasticsearch7.9
2G
- ELK 是 Elasticsearch、Logstash、Kibana 的简称,是近乎完美的开源实时日志分析平台
- 这三者是日志分析平台的核心组件,而并非全部
Elasticsearch
:实时全文搜索和分析引擎,提供搜集、分析、存储数据三大功能,具有分布式,零配置,自动发现,索引自动分片,索引副本机制,restful 风格接口,多数据源,自动搜索负载等特点
Logstash
:它支持几乎任何类型的日志,包括系统日志、错误日志和自定义应用程序日志,它可以从许多来源接收日志,这些来源包括 syslog、消息传递(例如 RabbitMQ)和 JMX,它能够以多种方式输出数据,包括电子邮件、websockets 和 Elasticsearch
Kibana:基于 Web 的图形界面,用于搜索、分析和可视化存储在 Elasticsearch 指标中的日志数据,它利用 Elasticsearch 的 REST 接口来检索数据,不仅允许用户创建他们自己的数据的定制仪表板视图,还允许他们以特殊的方式查询和过滤数据,Kibana 可以为 Logstash 和 Elasticsearch 提供友好的日志分析 web 界面,可以帮助你汇总、分析和搜索重要数据日志
配置 elk01 服务器
配置 elasticsearch
主机名 | 外网 IP | 内网 IP | 应用 | 内存 |
---|---|---|---|---|
lb01 | 10.0.0.5 | 172.16.1.5 | nginx、logstash7.9 | 2G |
lb02 | 10.0.0.6 | 172.16.1.6 | nginx、kibana7.9 | 2G |
web01 | 10.0.0.7 | 172.16.1.7 | nginx、filebeat7.9、kafka2.12 | 2G |
web02 | 10.0.0.8 | 172.16.1.8 | nginx、filebeat7.9、kafka2.12 | 2G |
elk01 | 10.0.0.81 | 172.16.1.81 | elasticsearch7.9 | 2G |
elk02 | 10.0.0.82 | 172.16.1.82 | elasticsearch7.9 | 2G |
Elasticsearch
:实时全文搜索和分析引擎,提供搜集、分析、存储数据三大功能,具有分布式,零配置,自动发现,索引自动分片,索引副本机制,restful 风格接口,多数据源,自动搜索负载等特点
Logstash
:它支持几乎任何类型的日志,包括系统日志、错误日志和自定义应用程序日志,它可以从许多来源接收日志,这些来源包括 syslog、消息传递(例如 RabbitMQ)和 JMX,它能够以多种方式输出数据,包括电子邮件、websockets 和 Elasticsearch
Kibana:基于 Web 的图形界面,用于搜索、分析和可视化存储在 Elasticsearch 指标中的日志数据,它利用 Elasticsearch 的 REST 接口来检索数据,不仅允许用户创建他们自己的数据的定制仪表板视图,还允许他们以特殊的方式查询和过滤数据,Kibana 可以为 Logstash 和 Elasticsearch 提供友好的日志分析 web 界面,可以帮助你汇总、分析和搜索重要数据日志
配置 elasticsearch
下载安装 elasticsearch
yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.9.1/elasticsearch-7.9.1-x86_64.rpm
编辑配置文件
sed -i 's/^[^#]/#&/' /etc/elasticsearch/elasticsearch.yml
cat >> /etc/elasticsearch/elasticsearch.yml << 'EOF'
cluster.name: elkstack
node.name: es01
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 127.0.0.1,10.0.0.81
http.port: 9200
discovery.seed_hosts: ["10.0.0.81","10.0.0.82"]
cluster.initial_master_nodes: ["10.0.0.81","10.0.0.82"]
EOF
解除 Elasticsearch 进程对内存锁定的限制
sed -i '/\[Service\]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/elasticsearch.service
重新加载 systemd 的配置文件
systemctl daemon-reload
启动服务并加入开机自启
systemctl start elasticsearch
systemctl enable elasticsearch
网页访问:10.0.0.81:9200
{
"name" : "es01",
"cluster_name" : "elkstack",
"cluster_uuid" : "BokyxBHiSzGckadvzfnVSQ",
"version" : {
"number" : "7.9.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",
"build_date" : "2020-09-01T21:22:21.964974Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
配置 elk02 服务器
配置 elasticsearch
下载安装 elasticsearch
yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.9.1/elasticsearch-7.9.1-x86_64.rpm
编辑配置文件
sed -i 's/^[^#]/#&/' /etc/elasticsearch/elasticsearch.yml
cat >> /etc/elasticsearch/elasticsearch.yml << 'EOF'
cluster.name: elkstack
node.name: es02
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 127.0.0.1,10.0.0.82
http.port: 9200
discovery.seed_hosts: ["10.0.0.81","10.0.0.82"]
cluster.initial_master_nodes: ["10.0.0.81","10.0.0.82"]
EOF
解除 Elasticsearch 进程对内存锁定的限制
sed -i '/\[Service\]/a LimitMEMLOCK=infinity' /usr/lib/systemd/system/elasticsearch.service
重新加载 systemd 的配置文件
systemctl daemon-reload
启动服务并加入开机自启
systemctl start elasticsearch
systemctl enable elasticsearch
网页访问:10.0.0.82:9200
{
"name" : "es02",
"cluster_name" : "elkstack",
"cluster_uuid" : "BokyxBHiSzGckadvzfnVSQ",
"version" : {
"number" : "7.9.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",
"build_date" : "2020-09-01T21:22:21.964974Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
检查集群状态
curl -X GET "localhost:9200/_cat/health?v"
配置 lb01 服务器
lb01:nginx 代理和 logstash 服务器
配置 nginx
配置官方 yum 源
cat > /etc/yum.repos.d/nginx.repo <<EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF
安装 Nginx 服务与其相关依赖
yum -y install gcc gcc-c++ autoconf pcre pcre-devel make automake wget httpd-tools vim tree nginx
创建 uid=666、gid=666 的用户和用户组
groupadd -g 666 www
useradd -u 666 -g 666 www -s /sbin/nologin -M
id www
修改 nginx 主配置文件中 user 为 www 用户
sed -i 's#user nginx;#user www;#g' /etc/nginx/nginx.conf
nginx 主配置文件增加 json 格式日志
### 编辑主配置文件(增加json格式日志)
vim /etc/nginx/nginx.conf
………………省略部分输出信息………………
access_log /var/log/nginx/access.log main;
# 在http层的原有日志下添加以下内容
log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status"}';
access_log /var/log/nginx/access_json.log access_json;
配置 proxy_params
cat > /etc/nginx/proxy_params <<EOF
proxy_set_header Host \$http_host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_connect_timeout 30;
proxy_send_timeout 60;
proxy_read_timeout 60;
proxy_buffering on;
proxy_buffer_size 32k;
proxy_buffers 4 128k;
EOF
配置负载均衡
cat > /etc/nginx/conf.d/web_proxy.conf <<EOF
upstream www_pools {
server 10.0.0.7;
server 10.0.0.8;
}
server {
listen 80;
server_name blog.xxx.com;
location / {
proxy_pass http://www_pools;
include proxy_params;
}
}
EOF
启动服务并加入开机自启
systemctl start nginx
systemctl enable nginx
配置 keepalived
安装 keepalived
yum -y install keepalived
编辑配置文件
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {
router_id keepalived01
}
# 每3秒执行一次脚本,脚本执行内容不能超过3秒,否则会中断再次重新执行脚本
vrrp_script check_web {
script "/etc/keepalived/scripts/check_web.sh"
interval 3
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 50
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.3
}
track_script {
check_web
}
}
EOF
创建脚本目录
mkdir -p /etc/keepalived/scripts
配置检测脚本
cat > /etc/keepalived/scripts/check_web.sh <<EOF
#!/bin/bash
nginx_status=\$(ps -ef|grep [n]ginx|wc -l)
if [ \$nginx_status -eq 0 ];then
systemctl stop keepalived
fi
EOF
检测脚本增加执行权限
chmod +x /etc/keepalived/scripts/check_web.sh
启动服务并设置开机自启
systemctl start keepalived
systemctl enable keepalived
配置 java 环境
安装 jdk11
yum -y install java-11-openjdk
查看 java_home
java -version
配置 logstash
下载安装 logstash
yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.9.1/logstash-7.9.1.rpm
授权 logstash 目录
chown -R logstash.logstash /usr/share/logstash/
启动服务并加入开机自启
systemctl start logstash
systemctl enable logstash
配置 logstash 环境变量
### 添加环境变量
cat > /etc/profile.d/logstash.sh <<EOF
export PATH="/usr/share/logstash/bin:\$PATH"
EOF
### 生效配置文件
source /etc/profile
检查进程
ps -ef | grep logstash
编辑配置文件
cat > /etc/logstash/conf.d/first-pipeline.conf <<EOF
input {
kafka {
type => "nginx_access_log"
codec => "json"
topics => ["nginx"]
decorate_events => true
bootstrap_servers => "10.0.0.7:9092,10.0.0.8:9092"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
mutate {
remove_field => ["message","input_type","@version","fields"]
}
geoip {
source => "clientip"
}
}
output {
stdout {}
if [type] == "nginx_access_log" {
elasticsearch {
index => "nginx-%{+YYYY.MM.dd}"
codec => "json"
hosts => ["10.0.0.81:9200","10.0.0.82:9200"]
}
}
}
EOF
启动 logstash
nohup logstash -f /etc/logstash/conf.d/first-pipeline.conf >/dev/null 2>&1 &
配置 lb02 服务器
lb02:nginx 代理和 kibana 服务器
配置 nginx
配置官方 yum 源
cat > /etc/yum.repos.d/nginx.repo <<EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF
安装 Nginx 服务与其相关依赖
yum -y install gcc gcc-c++ autoconf pcre pcre-devel make automake wget httpd-tools vim tree nginx
创建 uid=666、gid=666 的用户和用户组
groupadd -g 666 www
useradd -u 666 -g 666 www -s /sbin/nologin -M
id www
修改 nginx 主配置文件中 user 为 www 用户
sed -i 's#user nginx;#user www;#g' /etc/nginx/nginx.conf
nginx 主配置文件增加 json 格式日志
### 编辑主配置文件(增加json格式日志)
vim /etc/nginx/nginx.conf
………………省略部分输出信息………………
access_log /var/log/nginx/access.log main;
# 在http层的原有日志下添加以下内容
log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status"}';
access_log /var/log/nginx/access_json.log access_json;
配置 proxy_params
cat > /etc/nginx/proxy_params <<EOF
proxy_set_header Host \$http_host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_connect_timeout 30;
proxy_send_timeout 60;
proxy_read_timeout 60;
proxy_buffering on;
proxy_buffer_size 32k;
proxy_buffers 4 128k;
EOF
配置负载均衡
cat > /etc/nginx/conf.d/web_proxy.conf <<EOF
upstream www_pools {
server 10.0.0.7;
server 10.0.0.8;
}
server {
listen 80;
server_name blog.xxx.com;
location / {
proxy_pass http://www_pools;
include proxy_params;
}
}
EOF
代理 kibana
cat > /etc/nginx/conf.d/kibana.conf <<EOF
server {
listen 80;
server_name kibana.xxx.com;
location / {
proxy_pass http://10.0.0.6:5601;
include proxy_params;
}
}
EOF
启动服务并加入开机自启
systemctl start nginx
systemctl enable nginx
配置 keepalived
安装 keepalived
yum -y install keepalived
编辑配置文件
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {
router_id keepalived02
}
vrrp_instance VI_1 {
state BACKUP
nopreempt
interface eth0
virtual_router_id 50
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.3
}
}
EOF
启动服务并设置开机自启
systemctl start keepalived
systemctl enable keepalived
配置 kibana
下载安装 kibana
yum -y install https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.9.1/kibana-7.9.1-x86_64.rpm
编辑 kibana 配置文件
kibana_ip=$(ifconfig eth0 | awk 'NR==2 {print $2}')
es_ip=10.0.0.81
cat >> /etc/kibana/kibana.yml <<EOF
server.port: 5601
server.host: "${kibana_ip}"
elasticsearch.hosts: ["http://${es_ip}:9200"]
kibana.index: ".kibana"
i18n.locale: "zh-CN"
EOF
启动服务并加入开机自启
systemctl start kibana
systemctl enable kibana
检查进程
ps -ef | grep kibana
配置 wbe01 服务器
web01:nginx、filebeat、kafka 服务器
配置 nginx
配置官方 yum 源
cat > /etc/yum.repos.d/nginx.repo <<EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF
安装 Nginx 服务与其相关依赖
yum -y install gcc gcc-c++ autoconf pcre pcre-devel make automake wget httpd-tools vim tree nginx
创建 uid=666、gid=666 的用户和用户组
groupadd -g 666 www
useradd -u 666 -g 666 www -s /sbin/nologin -M
id www
修改 nginx 主配置文件中 user 为 www 用户
sed -i 's#user nginx;#user www;#g' /etc/nginx/nginx.conf
配置 php 第三方源
cat > /etc/yum.repos.d/php.repo <<EOF
[php-webtatic]
name = PHP Repository
baseurl = http://us-east.repo.webtatic.com/yum/el7/x86_64/
gpgcheck = 0
EOF
安装 php 及其相关依赖
yum -y install php71w php71w-cli php71w-common php71w-devel php71w-embedded php71w-gd php71w-mcrypt php71w-mbstring php71w-pdo php71w-xml php71w-fpm php71w-mysqlnd php71w-opcache php71w-pecl-memcached php71w-pecl-redis php71w-pecl-mongodb
修改 php-fpm 配置文件中 user 和 group 为 www 用户
sed -i 's#user = apache#user = www#g' /etc/php-fpm.d/www.conf
sed -i 's#group = apache#group = www#g' /etc/php-fpm.d/www.conf
编辑网站配置文件
cat > /etc/nginx/conf.d/www.conf <<EOF
server {
listen 80;
server_name blog.xxx.com;
location / {
root /code;
index index.html;
}
}
EOF
创建 nginx 站点目录
mkdir -p /code
编辑网站 html
echo "$HOSTNAME" > /code/index.html
nginx 主配置文件增加 json 格式日志
### 编辑主配置文件(增加json格式日志)
vim /etc/nginx/nginx.conf
………………省略部分输出信息………………
access_log /var/log/nginx/access.log main;
# 在http层的原有日志下添加以下内容
log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status"}';
access_log /var/log/nginx/access_json.log access_json;
启动服务并加入开机自启
systemctl start php-fpm nginx
systemctl enable php-fpm nginx
查看端口
netstat -lnupt | egrep "9000|80"
配置 filebeat
下载安装 filebeat7.9
yum -y install https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.9.1-x86_64.rpm
备份 filebeat 原配置文件
cp /etc/filebeat/filebeat.yml{,.bak}
注释配置文件
sed -i 's/^[^#]/#&/' /etc/filebeat/filebeat.yml
查看是否将配置文件全部注释
egrep -v "^$|^ *#" /etc/filebeat/filebeat.yml
编辑 filebeat 配置文件
cat >> /etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: \${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
# ------------------------------ kafka Output -------------------------------
output.kafka:
hosts: ["10.0.0.7:9092","10.0.0.8:9092"]
enabled: true
topic: 'nginx'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 100000000
processors:
- drop_event:
when:
regexp:
message: "^DBG:"
- drop_fields:
fields: ['input',"ecs.version"]
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
EOF
启动 nginx 模块
filebeat modules enable nginx
备份配置文件
cp /etc/filebeat/modules.d/nginx.yml{,.bak}
注释配置文件
sed -i 's/^[^#]/#&/' /etc/filebeat/modules.d/nginx.yml
查看是否将配置文件全部注释
egrep -v "^$|^ *#" /etc/filebeat/modules.d/nginx.yml
编辑 nginx 模块
cat >> /etc/filebeat/modules.d/nginx.yml <<EOF
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access_json.log"]
error:
enabled: true
ingress_controller:
enabled: false
EOF
启动服务并加入开机自启
systemctl start filebeat
systemctl enable filebeat
检查进程
ps -ef | grep filebeat
配置 java 环境
创建 app 目录
mkdir /app
上传安装包并解压
tar xf jdk-12.0.1_linux-x64_bin.tar.gz -C /app
做软连接
ln -s /app/jdk-12.0.1 /app/jdk
配置环境变量
### 编辑环境变量文件
cat > /etc/profile.d/jdk12.sh <<EOF
export JAVA_HOME=/app/jdk
export PATH=\$PATH:\$JAVA_HOME/bin
EOF
### 生效配置文件
source /etc/profile
配置 zookeeper
上传安装包并解压
tar xf zookeeper-3.4.14.tar.gz -C /app
做软连接
ln -s /app/zookeeper-3.4.14 /app/zookeeper
配置 kafka
上传安装包并解压
tar xf kafka_2.12-2.2.1.tgz -C /app
做软连接
ln -s /app/kafka_2.12-2.2.1 /app/kafka
备份配置文件
cp /app/kafka/config/zookeeper.properties{,.bak}
注释配置文件
sed -i 's/^[^#]/#&/' /app/kafka/config/zookeeper.properties
查看是否将配置文件全部注释
egrep -v "^$|^ *#" /app/kafka/config/zookeeper.properties
编辑配置文件
cat >> /app/kafka/config/zookeeper.properties <<EOF
# zk数据存放目录
dataDir=/app/kafka/data/zookeeper/data
# zk日志存放目录
dataLogDir=/app/kafka/data/zookeeper/logs
# 客户端连接zk服务的端口
clientPort=2181
# zk服务器之间或客户端与服务器之间维持心跳的时间间隔
ticktime=2000
# 允许follower连接并同步到Leader的初始化连接时间,当初始化连接时间超过该值,则表示连接失败
initLimit=20
# Leader与Follower之间发送消息时如果follower在设置时间内不能与leader通信,那么此follower将会被丢弃
syncLimit=10
# 以下IP信息根据配置kafka集群的服务器IP进行修改
# 2888是follower与leader交换信息的端口,3888是当leader挂了时用来执行选举时服务器相互通信的端口
server.1=10.0.0.7:2888:3888
server.2=10.0.0.8:2888:3888
EOF
创建 data/logs 目录
mkdir -p /app/kafka/data/zookeeper/{data,logs}
创建 myid 文件用于标识此服务器实例
echo 1 > /app/kafka/data/zookeeper/data/myid
备份配置文件
cp /app/kafka/config/server.properties{,.bak}
注释配置文件
sed -i 's/^[^#]/#&/' /app/kafka/config/server.properties
查看是否将配置文件全部注释
egrep -v "^$|^ *#" /app/kafka/config/server.properties
编辑配置文件
cat >> /app/kafka/config/server.properties <<EOF
broker.id=1
listeners=PLAINTEXT://10.0.0.7:9092
num.network.threads=3
num.io.threads=3
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/app/data/kafka/logs
num.partitions=6
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.0.0.7:2181,10.0.0.8:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
EOF
创建日志目录
mkdir -p /app/data/kafka/logs
修改 Zookeeper 的配置文件,首先进入安装路径 conf 目录,并将 zoo_sample.cfg 文件修改为 zoo.cfg,并对核心参数进行配置。
mv zoo_sample.cfg zoo.cfg
启动 zookeeper 服务
/app/zookeeper/bin/zkServer.sh start
查看端口
netstat -lnupt | grep 2181
启动 kafka 服务
nohup /app/kafka/bin/kafka-server-start.sh /app/kafka/config/server.properties &
查看启动的服务
jps -l
配置 wbe02 服务器
web01:nginx、filebeat、kafka 服务器
配置 nginx
配置官方 yum 源
cat > /etc/yum.repos.d/nginx.repo <<EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF
安装 Nginx 服务与其相关依赖
yum -y install gcc gcc-c++ autoconf pcre pcre-devel make automake wget httpd-tools vim tree nginx
创建 uid=666、gid=666 的用户和用户组
groupadd -g 666 www
useradd -u 666 -g 666 www -s /sbin/nologin -M
id www
修改 nginx 主配置文件中 user 为 www 用户
sed -i 's#user nginx;#user www;#g' /etc/nginx/nginx.conf
配置 php 第三方源
cat > /etc/yum.repos.d/php.repo <<EOF
[php-webtatic]
name = PHP Repository
baseurl = http://us-east.repo.webtatic.com/yum/el7/x86_64/
gpgcheck = 0
EOF
安装 php 及其相关依赖
yum -y install php71w php71w-cli php71w-common php71w-devel php71w-embedded php71w-gd php71w-mcrypt php71w-mbstring php71w-pdo php71w-xml php71w-fpm php71w-mysqlnd php71w-opcache php71w-pecl-memcached php71w-pecl-redis php71w-pecl-mongodb
修改 php-fpm 配置文件中 user 和 group 为 www 用户
sed -i 's#user = apache#user = www#g' /etc/php-fpm.d/www.conf
sed -i 's#group = apache#group = www#g' /etc/php-fpm.d/www.conf
编辑网站配置文件
cat > /etc/nginx/conf.d/www.conf <<EOF
server {
listen 80;
server_name blog.xxx.com;
location / {
root /code;
index index.html;
}
}
EOF
启动服务并加入开机自启
systemctl start php-fpm nginx
systemctl enable php-fpm nginx
查看端口
netstat -lnupt | egrep "9000|80"
创建 nginx 站点目录
mkdir -p /code
编辑网站 html
echo "$HOSTNAME" > /code/index.html
nginx 主配置文件增加 json 格式日志
### 编辑主配置文件(增加json格式日志)
vim /etc/nginx/nginx.conf
………………省略部分输出信息………………
access_log /var/log/nginx/access.log main;
# 在http层的原有日志下添加以下内容
log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status"}';
access_log /var/log/nginx/access_json.log access_json;
配置 filebeat
下载安装 filebeat7.9
yum -y install https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.9.1-x86_64.rpm
备份 filebeat 原配置文件
cp /etc/filebeat/filebeat.yml{,.bak}
注释配置文件
sed -i 's/^[^#]/#&/' /etc/filebeat/filebeat.yml
查看是否将配置文件全部注释
egrep -v "^$|^ *#" /etc/filebeat/filebeat.yml
编辑 filebeat 配置文件
cat >> /etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: \${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
# ------------------------------ kafka Output -------------------------------
output.kafka:
hosts: ["10.0.0.7:9092","10.0.0.8:9092"]
enabled: true
topic: 'nginx'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
processors:
- drop_event:
when:
regexp:
message: "^DBG:"
- drop_fields:
fields: ['input',"ecs.version"]
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
EOF
启动 nginx 模块
filebeat modules enable nginx
备份配置文件
cp /etc/filebeat/modules.d/nginx.yml{,.bak}
注释配置文件
sed -i 's/^[^#]/#&/' /etc/filebeat/modules.d/nginx.yml
查看是否将配置文件全部注释
egrep -v "^$|^ *#" /etc/filebeat/modules.d/nginx.yml
编辑 nginx 模块
cat >> /etc/filebeat/modules.d/nginx.yml <<EOF
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access_json.log"]
error:
enabled: true
ingress_controller:
enabled: false
EOF
启动服务并加入开机自启
systemctl start filebeat
systemctl enable filebeat
检查进程
ps -ef | grep filebeat
配置 java 环境
创建 app 目录
mkdir /app
上传安装包并解压
tar xf jdk-12.0.1_linux-x64_bin.tar.gz -C /app
做软连接
ln -s /app/jdk-12.0.1 /app/jdk
配置环境变量
### 编辑环境变量文件
cat > /etc/profile.d/jdk12.sh <<EOF
export JAVA_HOME=/app/jdk
export PATH=\$PATH:\$JAVA_HOME/bin
EOF
### 生效配置文件
source /etc/profile
配置 zookeeper
上传安装包并解压
tar xf zookeeper-3.4.14.tar.gz -C /app
做软连接
ln -s /app/zookeeper-3.4.14 /app/zookeeper
配置 kafka
上传安装包并解压
tar xf kafka_2.12-2.2.1.tgz -C /app
做软连接
ln -s /app/kafka_2.12-2.2.1 /app/kafka
备份配置文件
cp /app/kafka/config/zookeeper.properties{,.bak}
注释配置文件
sed -i 's/^[^#]/#&/' /app/kafka/config/zookeeper.properties
查看是否将配置文件全部注释
egrep -v "^$|^ *#" /app/kafka/config/zookeeper.properties
编辑配置文件
cat >> /app/kafka/config/zookeeper.properties <<EOF
# zk数据存放目录
dataDir=/app/kafka/data/zookeeper/data
# zk日志存放目录
dataLogDir=/app/kafka/data/zookeeper/logs
# 客户端连接zk服务的端口
clientPort=2181
# zk服务器之间或客户端与服务器之间维持心跳的时间间隔
ticktime=2000
# 允许follower连接并同步到Leader的初始化连接时间,当初始化连接时间超过该值,则表示连接失败
initLimit=20
# Leader与Follower之间发送消息时如果follower在设置时间内不能与leader通信,那么此follower将会被丢弃
syncLimit=10
# 以下IP信息根据配置kafka集群的服务器IP进行修改
# 2888是follower与leader交换信息的端口,3888是当leader挂了时用来执行选举时服务器相互通信的端口
server.1=10.0.0.7:2888:3888
server.2=10.0.0.8:2888:3888
EOF
创建 data/logs 目录
mkdir -p /app/kafka/data/zookeeper/{data,logs}
创建 myid 文件用于标识此服务器实例
echo 2 > /app/kafka/data/zookeeper/data/myid
备份配置文件
cp /app/kafka/config/server.properties{,.bak}
注释配置文件
sed -i 's/^[^#]/#&/' /app/kafka/config/server.properties
查看是否将配置文件全部注释
egrep -v "^$|^ *#" /app/kafka/config/server.properties
编辑配置文件
cat >> /app/kafka/config/server.properties <<EOF
# 修改broker
broker.id=2
# 修改本机
listeners=PLAINTEXT://10.0.0.8:9092
num.network.threads=3
num.io.threads=3
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/app/data/kafka/logs
num.partitions=6
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.0.0.7:2181,10.0.0.8:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
EOF
创建日志目录
mkdir -p /app/data/kafka/logs
启动 zookeeper 服务
nohup /app/kafka/bin/zookeeper-server-start.sh /app/kafka/config/zookeeper.properties >/dev/null 2>&1 &
查看端口
netstat -lnupt | grep 2181
启动 kafka 服务
nohup /app/kafka/bin/kafka-server-start.sh /app/kafka/config/server.properties >/dev/null 2>&1 &
查看启动的服务
jps -l
kafka 检查内容展示
# 查询所有主题
[root@web01 ~]# /app/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --list
__consumer_offsets
nginx
# 查询nginx主题详情
[root@web01 ~]# /app/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic nginx
Topic:nginx PartitionCount:6 ReplicationFactor:1 Configs:
Topic: nginx Partition: 0 Leader: 2 Replicas: 2 Isr: 2
Topic: nginx Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: nginx Partition: 2 Leader: 2 Replicas: 2 Isr: 2
Topic: nginx Partition: 3 Leader: 2 Replicas: 2 Isr: 2
Topic: nginx Partition: 4 Leader: 2 Replicas: 2 Isr: 2
Topic: nginx Partition: 5 Leader: 2 Replicas: 2 Isr: 2