An instance provided by ES consists of an ES cluster and a Kibana Console. The former can be accessed via the VIP address and port of your VPC, and the latter provides a public IP address for you to access from a browser. Currently, you can only ingest your data into the ES cluster on your own. The following describes how to import your logs into ES and access Kibana from a browser to perform query and analysis by taking the most typical log analysis architectures Filebeat + Elasticsearch + Kibana and Logstash + Elasticsearch + Kibana as examples.
Filebeat + Elasticsearch + Kibana
Deploying Filebeat
1. Download the Filebeat package and decompress it.
Note:
The Filebeat version should be compatible with the ES version.
2. Configure Filebeat
In this example, NGINX log is used as the input source, and the output is configured as the private VIP address and port of the ES cluster. If you use a Platinum Edition cluster, you need to add username and password to the output.
Enter the filebeat-6.4.3-linux-x86_64 directory and modify the filebeat.yml configuration file, as shown below:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
output.elasticsearch:
hosts: ["10.0.130.91:9200"]
protocol: "http"
username: "elastic"
password: "test"
3. Run Filebeat
In the filebeat-6.4.3-linux-x86_64 directory, run:
nohup ./filebeat -c filebeat.yml 2>&1>/dev/null &
Querying a log
1. On the cluster list page in the ES Console, select Operation > Kibana to enter the Kibana Console.
2. Go to Management > Index Patterns and add an index pattern named filebeat-6.4.3-*.
3. Click Discover and select the filebeat-6.4.3-* index to retrieve the NGINX access logs.
Logstash + Elasticsearch + Kibana
Environment preparations
You need to create one or more CVM instances as needed in the same VPC as the ES cluster and deploy the Logstash component on them.
A CVM instance should have at least 2 GB of memory.
You need to install Java 8 or above on the CVM instances.
Deploying Logstash
1. Download the Logstash package and decompress it.
Note:
The Logstash version should be compatible with the ES version.
2. Configure Logstash
In this example, NGINX log is used as the input source, and the output is configured as the private VIP address and port of the ES cluster. Create a test.conf configuration file, as shown below:
input {
file{
path =>"/var/log/nginx/access.log"# Path to the NGINX access log
start_position =>"beginning"# Read the log from the beginning of the file. If this parameter is not set, the log will be read when data is written to the file, just like `tail -f`
}
}
filter {
}
output {
elasticsearch {
hosts =>["http://172.16.0.145:9200"]# Private VIP address and port of the ES cluster
index =>"nginx_access-%{+YYYY.MM.dd}"# Index name. Indices are automatically created on a daily basis
user =>"elastic"# Username
password =>"yinan_test"# Password
}
}
The ES cluster is configured to automatically create an index by default. The nginx_access-%{+YYYY.MM.dd} index will be automatically created, and unless you need to set the mapping of the fields in the index in advance, you don't need to additionally call the API of ES to create indices.
3. Start Logstash
Enter the extracted logstash-6.4.3 directory of the Logstash package and run the following command to start Logstash in the background. Be sure to enter the path you create as the path to the configuration file.
[2019-05-29T12:20:27,125][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml'file because modules or command line options are specified
[2019-05-29T12:20:27,167][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"2e19b294-2b69-4da1-b87f-f4cb4a171b9c", :path=>"/root/logstash-6.4.3/data/uuid"}
[2019-05-29T12:20:30,871][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@10.0.130.91:10880/]}}
[2019-05-29T12:20:30,901][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@10.0.130.91:10880/, :path=>"/"}
[2019-05-29T12:20:31,449][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@10.0.130.91:10880/"}
[2019-05-29T12:20:31,567][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-05-29T12:20:31,574][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-05-29T12:20:31,670][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.0.130.91:10880"]}
[2019-05-29T12:20:31,749][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-05-29T12:20:32,094][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2019-05-29T12:20:33,242][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/root/logstash-6.4.3/data/plugins/inputs/file/.sincedb_d883144359d3b4f516b37dba51fab2a2", :path=>["/var/log/nginx/access.log"]}
[2019-05-29T12:20:33,329][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x12bdd65 run>"}
