当前位置:网站首页>Elasticsearch database | elasticsearch-7.5.0 application construction
Elasticsearch database | elasticsearch-7.5.0 application construction
2020-11-06 20:58:59 【mazhiline】
Elasticsearch Is a distributed and compliant RESTful Style search and data analysis engine .—— Elastic Stack Official website
build Elasticsearch Of “ Those things "
one day , On the deck of the golden Melly , Webb told Saab , It needs to be in the next project development process , Apply to Elasticsearch database , It is mainly used for data acquisition and real-time query of various devices on the golden plum , Maybe later, it will be used in the construction and application of distributed log system , Let Saab do a technology pre research first . therefore , Search a lot of information in Saab and find that , About Elasticsearch Database building , The information on the Internet is almost the same , Or most of them don't have a complete process . even to the extent that , Also found that , about Elasticsearch The practical application of database , Most pirates are limited to ELK And so on , It's like most pirates mention Redis, Just think this thing , Can only be used at the cache level , But in fact, people's functions are so powerful that they are beyond the imagination of the pirates . Even , Saab is in an Alibaba place , Found out about Elasticsearch Database free trial resources , But for free lunch , Maybe after that day's charge , You can only use one song “ Waves ” To celebrate the evil of gold . So in Saab careful research and analysis found that , In addition to the traditional deployment Elasticsearch Outside the database , There's a man named Docker Things that are , In addition to being able to operate stably Elasticsearch Outside the database , Also can save a lot of server memory . So , Saab began to explore in Docker To build Elasticsearch The road to ......
Docker Deploy Elasticsearch Of “ Those things "
1. Basic prerequisites
Alicloud or virtual machine VM Host a , And it's self-contained Docker Containers :
ps[️ matters needing attention ]: about Docker Please refer to :Linux The environment is based on CentOS7 Build deployment Docker Containers
2. Maximum number of open files
[2.1] Parameter optimization :ulimit -a |grep open
[[email protected] ~]# ulimit -a |grep open
open files (-n) 65535
[[email protected] ~]#
[2.2] Modify the configuration file : /etc/security/limits.conf
# Add the following two lines at the bottom
* soft nofile 65536
* hard nofile 65536
* soft nproc 2048
* hard nproc 4096
[2.3] Confirm whether to modify : ulimit -a |grep open
[[email protected] ~]# ulimit -a |grep open
open files (-n) 65535
[[email protected] ~]#
3. Adjust the number of system processes
Modification method :vim /etc/security/limits.d/20-nproc.conf
# Adjust to the following configuration
* soft nproc 4096
root soft nproc unlimited
4. Adjust virtual memory and maximum number of concurrent connections
Virtual memory (Virtual memory)->vm.max_map_count Elasticsearch By default hybrid mmapfs / niofs Table of contents to store indexes . The default operating system for mmap The count limit is too low , May cause an out of memory exception
[1] Temporary solution :
sysctl –w vm.max_map_count=262144
sysctl –w vm.max_map_count=262144
If you don't optimize, it will start Elasticsearch The following prompt will appear when :
max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
[2] Permanent effective plan : Update system files /etc/sysctl.conf Of vm.max_map_count Field Carry out orders :vim /etc/sysctl.conf
[[email protected] ~]# vim /etc/sysctl.conf
[[email protected] ~]#
#Settings Elasticsearch Virtual Memory—>655360
vm.max_map_count=655360
fs.file-max=655360
vm.overcommit_memory=1
Load into the system :sysctl -p --load /etc/sysctl.conf
[[email protected] elasticsearch]# sysctl -p --load /etc/sysctl.conf
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
vm.max_map_count = 655360
fs.file-max = 655360
vm.overcommit_memory = 1
[[email protected] elasticsearch]#
5. close elasticsearch Self test
stay elasticsearch.yml Add configuration item in the :
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
ps[️ matters needing attention ]: The main thing is to avoid the following problems : [1] Maximum number of open files : max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536] [2] Adjust virtual memory and maximum number of concurrent connections : max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [3] close elasticsearch Self test : system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk [4] Adjust the number of system processes : max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
Actual deployment ElasticSearch service
Docker The basic steps of deploying application services :Search[ Query image ]->Pull[ Pull the mirror image ]->Run[ Deploy the image ]
1. Inquire about Elasticsearch Mirror image :
docker search elasticsearch
ps[ matters needing attention ]:
- Generally, image resources are pulled from Docker Official warehouse [docker-hub] Pull , Or build it yourself Docker Cloud warehouse aliyun-docker etc.
- This tutorial selects ELK Images are based on ELK official Docker Warehouse elastic-io
2. Pull Elasticsearch Mirror image :
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.3.1
ps[ matters needing attention ]: 1. This tutorial uses 7.3.x edition , The latest version 7.4.x[ It mainly uses 7.3.x The version has been built in alicloud , Avoid the problem of getting into the pit ] 2. In the process of pulling, there may be [net/http: TLS handshake timeout] problem , Try it a few more times , The main problem is the limitation of network bandwidth
3. Change the image name :
docker tag docker.elastic.co/elasticsearch/elasticsearch:7.3.1 elasticsearch:latest
ps[ matters needing attention ]: 1. The name is too long to view , adopt docker tag source-image[ Source image ] target-image[ Mirror target ], It is recommended that [target-image:target-version] Format definition , And it doesn't take up space , It's like renaming a mirror image 2. For pulling kibana[docker.elastic.co/kibana/kibana:7.3.1] and logstash[docker.elastic.co/logstash/logstash:7.3.1] It is suggested to modify .
4. Deploy the mirror service : Deployment command :
docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms=512m -Xms=512m" elasticsearch:latest
Default configuration directory :
/usr/share/elasticsearch/config
/usr/share/elasticsearch/logs
Look at the container list :
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
![docker ps]
(https://ftp.bmp.ovh/imgs/2019/11/d7e09058af3fadbd.png) ps[ matters needing attention ]:
1. Need to open ports [9200 and 9300]->9200 As Http agreement , It is mainly used for external communication ,9300 As Tcp agreement ,jar Between is through tcp Protocol communication , Usually, you deploy a cluster by 9300 signal communication . recommend [ Host custom port :9200] 2.--restart=always : Configure container restart policy , When the host is rebooted, it is configured to boot automatically , You don't have to start it manually 3.--privileged: Configure container operation permissions [true-root Operating authority ,false- Current container user operation permissions ] 4. For deployment network mode, the default bridging mode is recommended , You can also customize it host Patterns, etc
5. Modify the configuration : Into the container :docker exec -it container-id[ Containers id] or container-name[ Container name ] /bin/bash for example :docker exec -it f2d2e97da375 /bin/bash #f2d2e97da375-> container-id
Modify the configuration file :
[[email protected] elasticsearch]# ls
LICENSE.txt NOTICE.txt README.textile bin config data jdk lib logs modules plugins
[[email protected] elasticsearch]#
[[email protected] elasticsearch]# cd config
[[email protected] config]# ls
elasticsearch.keystore elasticsearch.yml jvm.options log4j2.properties role_mapping.yml roles.yml users users_roles
[[email protected] config]# vi elasticsearch.yml
Add cross domain configuration :http.cors.enabled: true && http.cors.allow-origin: "*"
cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
And then quit exit Containers , Restart the container on the host :docker restart container-id[ Containers id] or container-name[ Container name ] docker restart f2d2e97da375
[[email protected] config]# exit
exit
[[email protected] ~]# docker restart f2d2e97da375
f2d2e97da375
[[email protected] ~]#
ps[ matters needing attention ]: 1. How to enter the container : Including the use of docker attach Order or docker exec command , Recommended docker exec command . reason :
- docker attach: Use exit Exit the container , Will cause the container to stop
- docker exec: Use exit Exit the container , Does not cause the container to stop
- Reference resources docker How to get into the container blog -docker Several ways to get into the container 2. If Docker Visual interface installed Portainer, It is recommended to enter the container in this way :
Build deployment ElasticSearch-Head service
ElasticSearch-Head: Elastic search cluster Web Front end interface , It's using Nodjs Built , It is mainly used to view ElasticSearch Related information
1. Pull Elasticsearch-Head Mirror image :docker pull mobz/elasticsearch-head:5
[[email protected] ~]# docker pull mobz/elasticsearch-head:5
5: Pulling from mobz/elasticsearch-head
75a822cd7888: Pull complete
57de64c72267: Pull complete
4306be1e8943: Pull complete
871436ab7225: Pull complete
0110c26a367a: Pull complete
1f04fe713f1b: Pull complete
723bac39028e: Pull complete
7d8cb47f1c60: Pull complete
7328dcf65c42: Pull complete
b451f2ccfb9a: Pull complete
304d5c28a4cf: Pull complete
4cf804850db1: Pull complete
Digest: sha256:55a3c82dd4ba776e304b09308411edd85de0dc9719f9d97a2f33baa320223f34
Status: Downloaded newer image for mobz/elasticsearch-head:5
docker.io/mobz/elasticsearch-head:5
[[email protected] ~]#
2. modify Elasticsearch-Head Image name :docker tag mobz/elasticsearch-head:5 elasticsearch-head:latest
[[email protected] ~]# docker tag mobz/elasticsearch-head:5 elasticsearch-head:latest
[[email protected] ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 05d1bcf30d16 7 days ago 207MB
nginx latest 540a289bab6c 3 weeks ago 126MB
prom/prometheus latest 2c8e464e47f4 3 weeks ago 129MB
moxm/sentinel-dashboard latest 0ccaac81584e 4 weeks ago 167MB
portainer latest 4cda95efb0e4 4 weeks ago 80.6MB
portainer/portainer latest 4cda95efb0e4 4 weeks ago 80.6MB
apache/skywalking-ui latest fa66ca9c9862 2 months ago 123MB
apache/skywalking-oap-server latest 376a37cdf65c 2 months ago 190MB
docker.elastic.co/kibana/kibana 7.3.1 b54865ba6b0b 2 months ago 1.01GB
docker.elastic.co/elasticsearch/elasticsearch 7.3.1 3d3aa92f641f 2 months ago 807MB
elasticsearch latest 3d3aa92f641f 2 months ago 807MB
prom/node-exporter latest e5a616e4b9cf 5 months ago 22.9MB
google/cadvisor latest eb1210707573 12 months ago 69.6MB
elasticsearch-head latest b19a5c98e43b 2 years ago 824MB
mobz/elasticsearch-head 5 b19a5c98e43b 2 years ago 824MB
tutum/influxdb latest c061e5808198 3 years ago 290MB
[[email protected] ~]#
3. Deploy Elasticsearch-Head Containers :
docker run -itd --restart=always --privileged=true -p 9100:9100 --name elasticsearch-head-server elasticsearch-head:latest
View Container Services :
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
4. Browser access :http://remote-ip:9100/
Actual test ElasticSearch-Head service
1. Prepare the data :
{
"companyId": "ebcb4f99e0cb4ad781278ae636c1031f",
"classifyName": " Hormone testing ",
"deviceType": " Hormone testing ",
"data": {
"deviceId": "20000060100000002",
"deviceNo": "QC00020000060100000002",
"appid": "00000000",
"sim": "89860404191792655118",
"csq": "30",
"electric": "98",
"voltage": "13279",
"softVer": "5143000500010601-01020006|5143000500010601-01010003",
"hardVer": "5143000500010601-00000002|5143000500010601-00000002",
"status": "0",
"date": "2020-09-13 11:23:52",
"elements": [
{
"key": "20",
"value": "10",
"time": "2020-09-13 11:23:52"
},
{
"key": "21",
"value": "11",
"time": "2020-09-13 11:23:52"
},
{
"key": "22",
"value": "12",
"time": "2020-09-13 11:23:52"
},
{
"key": "23",
"value": "13",
"time": "2020-09-13 11:23:52"
}
]
}
}
2. Set request header :application/json;charset=UTF-8
3. send out PUT request :http://remote-ip/device/deviceData/QC00020000060100000001
{
"_index": "device",
"_type": "deviceData",
"_id": "QC00020000060100000002",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}
4. Open the browser to see the results :
Build in real combat ElasticSearch summary
1. Deploy basic commands :
docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server --network-alias elasticsearch-server --hostname elasticsearch-server -v /docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /docker/elasticsearch/data:/usr/share/elasticsearch/data -v /docker/elasticsearch/logs:/usr/share/elasticsearch/logs -e "discovery.type=single-node" -e ES_JAVA_OPTS="-server -Xms512m -Xmx512m -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+AlwaysPreTouch -XX:-UseBiasedLocking " -e TZ="Asia/Shanghai" elasticsearch:latest
2. The configuration file [elasticsearch.yml ]:
cluster.name: elasticsearch-cluster
node.name: elasticsearch-server
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
transport.tcp.port: 9300
transport.tcp.compress: true
http.max_content_length: 128mb
3. Word segmentation plug-in installation :
./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.0/elasticsearch-analysis-ik-7.5.0.zip
4.Docker Deploy ElasticSearch-Head:
docker run -itd --restart=always --privileged=true -p 9100:9100 --network-alias elasticsearch-head-server --name elasticsearch-head-server --hostname elasticsearch-head-server -v /docker/elasticsearch-head/app:/usr/src/app elasticsearch-head:latest
5.ElasticSearch-Head Use 406 error : ajaxSettings part json To configure :
ajaxSettings: {
url: ajaxLocation,
isLocal: rlocalProtocol.test(ajaxLocParts[1]),
global: true,
type: "GET",
//contentType: "application/x-www-form-urlencoded",
contentType: "application/json;charset=UTF-8",
processData: true,
async: true,
/*
timeout: 0,
data: null,
dataType: null,
username: null,
password: null,
cache: null,
traditional: false,
headers: {},
*/
accepts: {
xml: "application/xml, text/xml",
html: "text/html",
text: "text/plain",
json: "application/json, text/javascript",
"*": "*/*",
},
inspectData part json To configure :
// Detect, normalize options and install callbacks for jsonp requests
jQuery.ajaxPrefilter("json jsonp", function (s, originalSettings, jqXHR) {
//var inspectData = s.contentType === "application/x-www-form-urlencoded" &&
var inspectData =
(s.contentType === "application/x-www-form-urlencoded" &&
typeof s.data === "string") ||
(s.contentType === "application/json;charset=UTF-8" &&
typeof s.data === "string");
Copyright notice : This article is an original blog article , Follow the relevant copyright agreement , If you reprint or share, please attach the original source link and link source .
版权声明
本文为[mazhiline]所创,转载请带上原文链接,感谢
边栏推荐
- C++ 数字、string和char*的转换
- C++学习——centos7上部署C++开发环境
- C++学习——一步步学会写Makefile
- C++学习——临时对象的产生与优化
- C++学习——对象的引用的用法
- C++编程经验(6):使用C++风格的类型转换
- Won the CKA + CKS certificate with the highest gold content in kubernetes in 31 days!
- C + + number, string and char * conversion
- C + + Learning -- capacity() and resize() in C + +
- C + + Learning -- about code performance optimization
猜你喜欢
-
C + + programming experience (6): using C + + style type conversion
-
Latest party and government work report ppt - Park ppt
-
在线身份证号码提取生日工具
-
Online ID number extraction birthday tool
-
️野指针?悬空指针?️ 一文带你搞懂!
-
Field pointer? Dangling pointer? This article will help you understand!
-
HCNA Routing&Switching之GVRP
-
GVRP of hcna Routing & Switching
-
Seq2Seq实现闲聊机器人
-
【闲聊机器人】seq2seq模型的原理
随机推荐
- LeetCode 91. 解码方法
- Seq2seq implements chat robot
- [chat robot] principle of seq2seq model
- Leetcode 91. Decoding method
- HCNA Routing&Switching之GVRP
- GVRP of hcna Routing & Switching
- HDU7016 Random Walk 2
- [Code+#1]Yazid 的新生舞会
- CF1548C The Three Little Pigs
- HDU7033 Typing Contest
- HDU7016 Random Walk 2
- [code + 1] Yazid's freshman ball
- CF1548C The Three Little Pigs
- HDU7033 Typing Contest
- Qt Creator 自动补齐变慢的解决
- HALCON 20.11:如何处理标定助手品质问题
- HALCON 20.11:标定助手使用注意事项
- Solution of QT creator's automatic replenishment slowing down
- Halcon 20.11: how to deal with the quality problem of calibration assistant
- Halcon 20.11: precautions for use of calibration assistant
- “十大科学技术问题”揭晓!|青年科学家50²论坛
- "Top ten scientific and technological issues" announced| Young scientists 50 ² forum
- 求反转链表
- Reverse linked list
- js的数据类型
- JS data type
- 记一次文件读写遇到的bug
- Remember the bug encountered in reading and writing a file
- 单例模式
- Singleton mode
- 在这个 N 多编程语言争霸的世界,C++ 究竟还有没有未来?
- In this world of N programming languages, is there a future for C + +?
- es6模板字符
- js Promise
- js 数组方法 回顾
- ES6 template characters
- js Promise
- JS array method review
- 【Golang】️走进 Go 语言️ 第一课 Hello World
- [golang] go into go language lesson 1 Hello World