Elasticsearch database | elasticsearch-7.5.0 application construction

Elasticsearch Is a distributed and compliant RESTful Style search and data analysis engine .—— Elastic Stack Official website

build Elasticsearch Of “ Those things "

one day , On the deck of the golden Melly , Webb told Saab , It needs to be in the next project development process , Apply to Elasticsearch database , It is mainly used for data acquisition and real-time query of various devices on the golden plum , Maybe later, it will be used in the construction and application of distributed log system , Let Saab do a technology pre research first . therefore , Search a lot of information in Saab and find that , About Elasticsearch Database building , The information on the Internet is almost the same , Or most of them don't have a complete process . even to the extent that , Also found that , about Elasticsearch The practical application of database , Most pirates are limited to ELK And so on , It's like most pirates mention Redis, Just think this thing , Can only be used at the cache level , But in fact, people's functions are so powerful that they are beyond the imagination of the pirates . Even , Saab is in an Alibaba place , Found out about Elasticsearch Database free trial resources , But for free lunch , Maybe after that day's charge , You can only use one song “ Waves ” To celebrate the evil of gold . So in Saab careful research and analysis found that , In addition to the traditional deployment Elasticsearch Outside the database , There's a man named Docker Things that are , In addition to being able to operate stably Elasticsearch Outside the database , Also can save a lot of server memory . So , Saab began to explore in Docker To build Elasticsearch The road to ......

Docker Deploy Elasticsearch Of “ Those things "

1. Basic prerequisites

Alicloud or virtual machine VM Host a , And it's self-contained Docker Containers ：

ps[️ matters needing attention ]： about Docker Please refer to ：Linux The environment is based on CentOS7 Build deployment Docker Containers

2. Maximum number of open files

[2.1] Parameter optimization ：ulimit -a |grep open

[[email protected] ~]# ulimit -a |grep open
open files   (-n) 65535
[[email protected] ~]# 

[2.2] Modify the configuration file ： /etc/security/limits.conf

# Add the following two lines at the bottom 
* soft nofile 65536
* hard nofile 65536
* soft nproc 2048
* hard nproc 4096

[2.3] Confirm whether to modify ： ulimit -a |grep open

[[email protected] ~]# ulimit -a |grep open
open files   (-n) 65535
[[email protected] ~]# 

3. Adjust the number of system processes

Modification method ：vim /etc/security/limits.d/20-nproc.conf

# Adjust to the following configuration 
*          soft    nproc     4096
root       soft    nproc     unlimited

4. Adjust virtual memory and maximum number of concurrent connections

Virtual memory （Virtual memory）->vm.max_map_count Elasticsearch By default hybrid mmapfs / niofs Table of contents to store indexes . The default operating system for mmap The count limit is too low , May cause an out of memory exception

[1] Temporary solution ：

sysctl –w vm.max_map_count=262144
sysctl –w vm.max_map_count=262144

If you don't optimize, it will start Elasticsearch The following prompt will appear when ：

max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]

[2] Permanent effective plan ： Update system files /etc/sysctl.conf Of vm.max_map_count Field Carry out orders ：vim /etc/sysctl.conf

[[email protected] ~]# vim /etc/sysctl.conf
[[email protected] ~]#
#Settings Elasticsearch Virtual Memory—>655360
vm.max_map_count=655360
fs.file-max=655360
vm.overcommit_memory=1

Load into the system ：sysctl -p --load /etc/sysctl.conf

[[email protected] elasticsearch]# sysctl -p --load /etc/sysctl.conf
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
vm.max_map_count = 655360
fs.file-max = 655360
vm.overcommit_memory = 1
[[email protected] elasticsearch]#

5. close elasticsearch Self test

stay elasticsearch.yml Add configuration item in the ：

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

ps[️ matters needing attention ]： The main thing is to avoid the following problems ： [1] Maximum number of open files : max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536] [2] Adjust virtual memory and maximum number of concurrent connections : max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [3] close elasticsearch Self test : system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk [4] Adjust the number of system processes : max number of threads [1024] for user [lish] likely too low, increase to at least [2048]

Actual deployment ElasticSearch service

Docker The basic steps of deploying application services ：Search[ Query image ]->Pull[ Pull the mirror image ]->Run[ Deploy the image ]

1. Inquire about Elasticsearch Mirror image ：

docker search elasticsearch 

ps[ matters needing attention ]：

1. Generally, image resources are pulled from Docker Official warehouse [docker-hub] Pull , Or build it yourself Docker Cloud warehouse aliyun-docker etc.
2. This tutorial selects ELK Images are based on ELK official Docker Warehouse elastic-io

2. Pull Elasticsearch Mirror image ：

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.3.1

ps[ matters needing attention ]： 1. This tutorial uses 7.3.x edition , The latest version 7.4.x[ It mainly uses 7.3.x The version has been built in alicloud , Avoid the problem of getting into the pit ] 2. In the process of pulling, there may be [net/http: TLS handshake timeout] problem , Try it a few more times , The main problem is the limitation of network bandwidth

3. Change the image name :

docker tag docker.elastic.co/elasticsearch/elasticsearch:7.3.1  elasticsearch:latest

ps[ matters needing attention ]： 1. The name is too long to view , adopt docker tag source-image[ Source image ] target-image[ Mirror target ], It is recommended that [target-image:target-version] Format definition , And it doesn't take up space , It's like renaming a mirror image 2. For pulling kibana[docker.elastic.co/kibana/kibana:7.3.1] and logstash[docker.elastic.co/logstash/logstash:7.3.1] It is suggested to modify .

4. Deploy the mirror service ： Deployment command ：

docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms=512m -Xms=512m" elasticsearch:latest

Default configuration directory ：

/usr/share/elasticsearch/config
/usr/share/elasticsearch/logs

Look at the container list ：

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
![docker ps]

(https://ftp.bmp.ovh/imgs/2019/11/d7e09058af3fadbd.png) ps[ matters needing attention ]:

1. Need to open ports [9200 and 9300]->9200 As Http agreement , It is mainly used for external communication ,9300 As Tcp agreement ,jar Between is through tcp Protocol communication , Usually, you deploy a cluster by 9300 signal communication . recommend [ Host custom port :9200] 2.--restart=always : Configure container restart policy , When the host is rebooted, it is configured to boot automatically , You don't have to start it manually 3.--privileged： Configure container operation permissions [true-root Operating authority ,false- Current container user operation permissions ] 4. For deployment network mode, the default bridging mode is recommended , You can also customize it host Patterns, etc

5. Modify the configuration ： Into the container ：docker exec -it container-id[ Containers id] or container-name[ Container name ] /bin/bash for example ：docker exec -it f2d2e97da375 /bin/bash #f2d2e97da375-> container-id

Modify the configuration file :

[[email protected] elasticsearch]# ls 
LICENSE.txt  NOTICE.txt  README.textile  bin  config  data  jdk  lib  logs  modules  plugins
[[email protected] elasticsearch]# 
[[email protected] elasticsearch]# cd config  
[[email protected] config]# ls
elasticsearch.keystore  elasticsearch.yml  jvm.options  log4j2.properties  role_mapping.yml  roles.yml  users  users_roles
[[email protected] config]# vi elasticsearch.yml 

Add cross domain configuration ：http.cors.enabled: true && http.cors.allow-origin: "*"

cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"

And then quit exit Containers , Restart the container on the host ：docker restart container-id[ Containers id] or container-name[ Container name ] docker restart f2d2e97da375

[[email protected] config]# exit
exit
[[email protected] ~]# docker restart f2d2e97da375
f2d2e97da375
[[email protected] ~]# 

ps[ matters needing attention ]： 1. How to enter the container ： Including the use of  docker attach  Order or  docker exec  command , Recommended  docker exec  command . reason ：

• docker attach: Use exit Exit the container , Will cause the container to stop
• docker exec： Use exit Exit the container , Does not cause the container to stop
• Reference resources docker How to get into the container blog -docker Several ways to get into the container 2. If Docker Visual interface installed  Portainer, It is recommended to enter the container in this way ：

Build deployment ElasticSearch-Head service

ElasticSearch-Head： Elastic search cluster Web Front end interface , It's using Nodjs Built , It is mainly used to view ElasticSearch Related information

1. Pull Elasticsearch-Head Mirror image ：docker pull mobz/elasticsearch-head:5

[[email protected] ~]# docker pull mobz/elasticsearch-head:5
5: Pulling from mobz/elasticsearch-head
75a822cd7888: Pull complete 
57de64c72267: Pull complete 
4306be1e8943: Pull complete 
871436ab7225: Pull complete 
0110c26a367a: Pull complete 
1f04fe713f1b: Pull complete 
723bac39028e: Pull complete 
7d8cb47f1c60: Pull complete 
7328dcf65c42: Pull complete 
b451f2ccfb9a: Pull complete 
304d5c28a4cf: Pull complete 
4cf804850db1: Pull complete 
Digest: sha256:55a3c82dd4ba776e304b09308411edd85de0dc9719f9d97a2f33baa320223f34
Status: Downloaded newer image for mobz/elasticsearch-head:5
docker.io/mobz/elasticsearch-head:5
[[email protected] ~]# 

2. modify Elasticsearch-Head Image name ：docker tag  mobz/elasticsearch-head:5   elasticsearch-head:latest

[[email protected] ~]# docker tag  mobz/elasticsearch-head:5        elasticsearch-head:latest
[[email protected] ~]# docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
grafana/grafana                                 latest              05d1bcf30d16        7 days ago          207MB
nginx                                           latest              540a289bab6c        3 weeks ago         126MB
prom/prometheus                                 latest              2c8e464e47f4        3 weeks ago         129MB
moxm/sentinel-dashboard                         latest              0ccaac81584e        4 weeks ago         167MB
portainer                                       latest              4cda95efb0e4        4 weeks ago         80.6MB
portainer/portainer                             latest              4cda95efb0e4        4 weeks ago         80.6MB
apache/skywalking-ui                            latest              fa66ca9c9862        2 months ago        123MB
apache/skywalking-oap-server                    latest              376a37cdf65c        2 months ago        190MB
docker.elastic.co/kibana/kibana                 7.3.1               b54865ba6b0b        2 months ago        1.01GB
docker.elastic.co/elasticsearch/elasticsearch   7.3.1               3d3aa92f641f        2 months ago        807MB
elasticsearch                                   latest              3d3aa92f641f        2 months ago        807MB
prom/node-exporter                              latest              e5a616e4b9cf        5 months ago        22.9MB
google/cadvisor                                 latest              eb1210707573        12 months ago       69.6MB
elasticsearch-head                              latest              b19a5c98e43b        2 years ago         824MB
mobz/elasticsearch-head                         5                   b19a5c98e43b        2 years ago         824MB
tutum/influxdb                                  latest              c061e5808198        3 years ago         290MB
[[email protected] ~]# 

3. Deploy Elasticsearch-Head Containers ：

docker run -itd --restart=always --privileged=true -p 9100:9100 --name elasticsearch-head-server elasticsearch-head:latest

View Container Services ：

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"

4. Browser access ：http://remote-ip:9100/

Actual test ElasticSearch-Head service

1. Prepare the data ：

{
    "companyId": "ebcb4f99e0cb4ad781278ae636c1031f",
    "classifyName": " Hormone testing ",
    "deviceType": " Hormone testing ",
    "data": {
    	"deviceId": "20000060100000002",
        "deviceNo": "QC00020000060100000002",
        "appid": "00000000",
        "sim": "89860404191792655118",
        "csq": "30",
        "electric": "98",
        "voltage": "13279",
        "softVer": "5143000500010601-01020006|5143000500010601-01010003",
        "hardVer": "5143000500010601-00000002|5143000500010601-00000002",
        "status": "0",
        "date": "2020-09-13 11:23:52",
        "elements": [
        	{
                "key": "20",
                "value": "10",
                "time": "2020-09-13 11:23:52"
            },
            {
                "key": "21",
                "value": "11",
                "time": "2020-09-13 11:23:52"
            },
            {
                "key": "22",
                "value": "12",
                "time": "2020-09-13 11:23:52"
            },
            {
                "key": "23",
                "value": "13",
                "time": "2020-09-13 11:23:52"
            }
        ]
    }
}

2. Set request header ：application/json;charset=UTF-8

3. send out PUT request ：http://remote-ip/device/deviceData/QC00020000060100000001

{
    "_index": "device",
    "_type": "deviceData",
    "_id": "QC00020000060100000002",
    "_version": 1,
    "result": "created",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "created": true
}

4. Open the browser to see the results ：

Build in real combat ElasticSearch summary

1. Deploy basic commands ：

docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server --network-alias elasticsearch-server --hostname elasticsearch-server -v /docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /docker/elasticsearch/data:/usr/share/elasticsearch/data -v /docker/elasticsearch/logs:/usr/share/elasticsearch/logs  -e "discovery.type=single-node" -e ES_JAVA_OPTS="-server -Xms512m -Xmx512m -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+AlwaysPreTouch -XX:-UseBiasedLocking " -e TZ="Asia/Shanghai"  elasticsearch:latest

2. The configuration file [elasticsearch.yml ]:

cluster.name: elasticsearch-cluster
node.name: elasticsearch-server
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
transport.tcp.port: 9300
transport.tcp.compress: true
http.max_content_length: 128mb

3. Word segmentation plug-in installation ：

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.0/elasticsearch-analysis-ik-7.5.0.zip

4.Docker Deploy ElasticSearch-Head：

docker run -itd --restart=always --privileged=true -p 9100:9100 --network-alias elasticsearch-head-server --name elasticsearch-head-server --hostname elasticsearch-head-server -v /docker/elasticsearch-head/app:/usr/src/app elasticsearch-head:latest

5.ElasticSearch-Head Use 406 error ： ajaxSettings part json To configure ：

    ajaxSettings: {
      url: ajaxLocation,
      isLocal: rlocalProtocol.test(ajaxLocParts[1]),
      global: true,
      type: "GET",
      //contentType: "application/x-www-form-urlencoded",
      contentType: "application/json;charset=UTF-8",
      processData: true,
      async: true,
      /*
		timeout: 0,
		data: null,
		dataType: null,
		username: null,
		password: null,
		cache: null,
		traditional: false,
		headers: {},
		*/

      accepts: {
        xml: "application/xml, text/xml",
        html: "text/html",
        text: "text/plain",
        json: "application/json, text/javascript",
        "*": "*/*",
      },

inspectData part json To configure ：

  // Detect, normalize options and install callbacks for jsonp requests
  jQuery.ajaxPrefilter("json jsonp", function (s, originalSettings, jqXHR) {
    //var inspectData = s.contentType === "application/x-www-form-urlencoded" &&
    var inspectData =
      (s.contentType === "application/x-www-form-urlencoded" &&
        typeof s.data === "string") ||
      (s.contentType === "application/json;charset=UTF-8" &&
        typeof s.data === "string");

Copyright notice ： This article is an original blog article , Follow the relevant copyright agreement , If you reprint or share, please attach the original source link and link source .