搭建云上日志收集分析系统（二）_服务革新_亚马逊云科技 (Amazon Web Services）

QCon 演讲火热征集中，快来分享技术实践与洞见！了解详情 



 写点什么

背景说明

应用系统的日志收集与分析工作对运维来说至关重要。常见的系统解决方案中开源技术栈 ELK（Elastic Stack: Elasticsearch, Logstash, Kibana）是当前比较流行的选择。下面我们会讨论另一种构建于云原生设计的类似于 ELK 的解决方案 EKK（Amazon Elasticsearch Service, Amazon Kinesis, and Kibana）。

EKK 的优势在于组件是 AWS 托管服务，不必自己安装、运维，并且与 AWS 的其它服务轻松集成，可以很轻松的部署一套可靠、可扩展、安全、容错以及解耦和基于事件的解决方案。

传统的 Elasticsearch 中，日志数据的不断膨胀，对数据的生命周期管理越来越重要（应对此需求的新功能 ILM(index lifecycle management)在 Elasticsearch 7.0 中闪亮登场）。本文不介绍 ILM，介绍另一种解决方案：使用 Lambda 配合实现数据的轮换。

配置需要收集日志服务器

1. 在 EC2 服务中为需要收集日志的 EC2 分配角色

分配角色

Linux 日志环境

安装 Agent

$sudo yum install –y aws-kinesis-agent

或者

$sudo yum install –y https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm

打开并编辑配置文件：/etc/aws-kinesis/agent.json

{
"cloudwatch.emitMetrics": true,
"firehose.endpoint": "firehose.cn-north-1.amazonaws.com.cn",
"flows": [
{
"filePattern": "/var/log/httpd/access_log*",
"deliveryStream": "EKK-LogFirehose-apachelog",
"dataProcessingOptions": [
{
"optionName": "LOGTOJSON",
"logFormat": "COMMONAPACHELOG"
}
]


}
]
}
$sudo service aws-kinesis-agent start
$sudo chkconfig aws-kinesis-agent on

复制代码

2. Windows 日志环境

下载安装 Agent

https://s3-us-west-2.amazonaws.com/kinesis-agent-windows/downloads/AWSKinesisTap.1.1.168.1.msi

C:\Program Files\Amazon\AWSKinesisTap\appsettings.json

{
"Sources": [
{
"Id": "PerformanceCounter",
"SourceType": "WindowsPerformanceCounterSource",
"Categories": [
{
"Category": "Server",
"Counters": [
"Files Open",
"Logon Total"
]
},
{
"Category": "LogicalDisk",
"Instances": "*",
"Counters": [
"% Free Space",
{
"Counter": "Disk Reads/sec",
"Unit": "Count/Second"
}
]
}
],
}
],
"Sinks": [
{
"Namespace": "MyServiceMetrics",
"Region": "cn-north-1",
"Id": "CloudWatchSink",
"SinkType": "CloudWatch"
},
{
"Id": "WindowsLogKinesisFirehoseSink",
"SinkType": "KinesisFirehose",
"StreamName": "EKK-LogFirehose-iislog",
"Region": "cn-north-1",
"QueueType": "file"
}
],
"Pipes": [
{
"Id": "PerformanceCounterToCloudWatch",
"SourceRef": "PerformanceCounter",
"SinkRef": "WindowsLogKinesisFirehoseSink"
}
]
}

复制代码

参考： https://docs.aws.amazon.com/zh_cn/kinesis-agent-windows/latest/userguide/getting-started.html

【可选】模拟一个 Apache Log 环境为后面的测试验证

在 EC2 服务中创建一个 Linux EC2

chmod 400 EKK-test.pem

ssh -i EKK-test.pem ec2-user@52.81.85.86

安装 Kinesis Agent

$sudo yum install –y https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm

配置 Agent

$sudo vi /etc/aws-kinesis/agent.json

{
"cloudwatch.emitMetrics": true,
"firehose.endpoint": "firehose.cn-north-1.amazonaws.com.cn",
"flows": [
{
"filePattern": "/var/log/httpd/access_log*",
"deliveryStream": "EKK-LogFirehose-apachelog",
"dataProcessingOptions": [
{
"optionName": "LOGTOJSON",
"logFormat": "COMMONAPACHELOG"
}
]


}
]
}

复制代码

$sudo service aws-kinesis-agent start

$sudo chkconfig aws-kinesis-agent on

$sudo mkdir /var/log/httpd

安装 Fake 日志生成程序

参考 https://github.com/kiritbasu/Fake-Apache-Log-Generator

$sudo yum install -y git

$git clone https://github.com/kiritbasu/Fake-Apache-Log-Generator.git

$sudo yum install python-pip -y

$cd Fake-Apache-Log-Generator/

$sudo pip install -r requirements.txt

产生日志脚本

$cd ~

$sudo vi test.sh

#!/bin/bash# chkconfig: 2345 10 90cd /var/log/httpd/while truedosudo python /home/ec2-user/Fake-Apache-Log-Generator/apache-fake-log-gen.py -n 100 -o LOGsleep 10done

后台运行脚本：

$sudo sh ./test.sh &

Fake 日志产生在目录 /var/log/httpd 中。