背景说明

应用系统的日志收集与分析工作对运维来说至关重要。常见的系统解决方案中开源技术栈ELK（Elastic Stack: Elasticsearch, Logstash, Kibana）是当前比较流行的选择。下面我们会讨论另一种构建于云原生设计的类似于ELK的解决方案EKK（Amazon Elasticsearch Service, Amazon Kinesis, and Kibana）。

EKK的优势在于组件是AWS托管服务，不必自己安装、运维，并且与AWS的其它服务轻松集成，可以很轻松的部署一套可靠、可扩展、安全、容错以及解耦和基于事件的解决方案。

传统的Elasticsearch中，日志数据的不断膨胀，对数据的生命周期管理越来越重要（应对此需求的新功能ILM(index lifecycle management)在Elasticsearch 7.0中闪亮登场）。本文不介绍ILM，介绍另一种解决方案：使用Lambda配合实现数据的轮换。

配置需要收集日志服务器

1. 在EC2服务中为需要收集日志的EC2分配角色

分配角色

Linux日志环境

安装Agent

$sudo yum install –y aws-kinesis-agent

或者

$sudo yum install –y https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm

打开并编辑配置文件：/etc/aws-kinesis/agent.json

{

"cloudwatch.emitMetrics": true,

"firehose.endpoint": "firehose.cn-north-1.amazonaws.com.cn",

"flows": [

{

"filePattern": "/var/log/httpd/access_log*",

"deliveryStream": "EKK-LogFirehose-apachelog",

"dataProcessingOptions": [

{

"optionName": "LOGTOJSON",

"logFormat": "COMMONAPACHELOG"

}

]



}

]

}

$sudo service aws-kinesis-agent start

$sudo chkconfig aws-kinesis-agent on

2. Windows日志环境

下载安装Agent

https://s3-us-west-2.amazonaws.com/kinesis-agent-windows/downloads/AWSKinesisTap.1.1.168.1.msi

C:\Program Files\Amazon\AWSKinesisTap\appsettings.json

{

"Sources": [

{

"Id": "PerformanceCounter",

"SourceType": "WindowsPerformanceCounterSource",

"Categories": [

{

"Category": "Server",

"Counters": [

"Files Open",

"Logon Total"

]

},

{

"Category": "LogicalDisk",

"Instances": "*",

"Counters": [

"% Free Space",

{

"Counter": "Disk Reads/sec",

"Unit": "Count/Second"

}

]

}

],

}

],

"Sinks": [

{

"Namespace": "MyServiceMetrics",

"Region": "cn-north-1",

"Id": "CloudWatchSink",

"SinkType": "CloudWatch"

},

{

"Id": "WindowsLogKinesisFirehoseSink",

"SinkType": "KinesisFirehose",

"StreamName": "EKK-LogFirehose-iislog",

"Region": "cn-north-1",

"QueueType": "file"

}

],

"Pipes": [

{

"Id": "PerformanceCounterToCloudWatch",

"SourceRef": "PerformanceCounter",

"SinkRef": "WindowsLogKinesisFirehoseSink"

}

]

}

参考： https://docs.aws.amazon.com/zh_cn/kinesis-agent-windows/latest/userguide/getting-started.html

【可选】模拟一个Apache Log环境为后面的测试验证

在EC2服务中创建一个Linux EC2

chmod 400 EKK-test.pem

ssh -i EKK-test.pem ec2-user@52.81.85.86

安装Kinesis Agent

$sudo yum install –y https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm

配置Agent

$sudo vi /etc/aws-kinesis/agent.json

{

"cloudwatch.emitMetrics": true,

"firehose.endpoint": "firehose.cn-north-1.amazonaws.com.cn",

"flows": [

{

"filePattern": "/var/log/httpd/access_log*",

"deliveryStream": "EKK-LogFirehose-apachelog",

"dataProcessingOptions": [

{

"optionName": "LOGTOJSON",

"logFormat": "COMMONAPACHELOG"

}

]



}

]

}

$sudo service aws-kinesis-agent start

$sudo chkconfig aws-kinesis-agent on

$sudo mkdir /var/log/httpd

安装Fake日志生成程序

参考 https://github.com/kiritbasu/Fake-Apache-Log-Generator

$sudo yum install -y git

$git clone https://github.com/kiritbasu/Fake-Apache-Log-Generator.git

$sudo yum install python-pip -y

$cd Fake-Apache-Log-Generator/

$sudo pip install -r requirements.txt

产生日志脚本

$cd ~

$sudo vi test.sh

#!/bin/bash# chkconfig: 2345 10 90cd /var/log/httpd/while truedosudo python /home/ec2-user/Fake-Apache-Log-Generator/apache-fake-log-gen.py -n 100 -o LOGsleep 10done

后台运行脚本：

$sudo sh ./test.sh &

Fake日志产生在目录 /var/log/httpd 中。

【可选】可使用Systems Manager（SSM）安装Agent

1. 为EC2的Role EKK-EC2 附加SSM需要的权限

2. 使用SSM服务

Linux 可使用AWS-RunShellScript

https://s3.cn-north-1.amazonaws.com.cn/awschinablog/picture replacement4.jpg)
运行Shell的目标EC2 可直接选择，或者按Tag筛选

Windows 类似，可使用AWS-RunPowerShellScript 运行powershell script

https://s3.cn-north-1.amazonaws.com.cn/awschinablog/Amazon Elasticsearch Service, Amazon Kinesis, and Kibana39.jpg">

作者简介

陈朕，AWS解决方案架构师，负责基于AWS云计算方案架构的咨询和设计，在国内推广AWS云平台技术和各种解决方案。十余年分布式应用、大数据的分布式处理经验。

本文转载自AWS技术博客

原文链接：

https://amazonaws-china.com/cn/blogs/china/ekk-amazon-elasticsearch-service-amazon-kinesis-and-kibana/

创作场景

搭建云上日志收集分析系统（二）