服务器部署 OpenSearch 集群

发表于 上层目录 ， 阅读次数：

Docker Compose 部署 Opensearch 集群并启用 Security Plugin

  • Opensearch 3.5.0

在每台节点机器上使用本文档的目录结构，通过 Docker Compose 以 Host Network 模式部署 OpenSearch 3 节点集群，并启用 OpenSearch Security + TLS（HTTP 与 Transport 双层 TLS）。

规划如下信息如下：

  • opensearch-1：172.16.10.72 ,配置 32 vCPU， 32G RAM

  • opensearch-2: 172.16.10.70

  • opensearch-3: 172.16.10.71

  • 集群名：opensearch-prod

  • 角色 ：3 台节点都同时作为

    • cluster-manager
    • data
    • ingest

    这样 3 节点具备仲裁能力，任意挂 1 台仍可工作。

  • Docker Host Network 模式会直接占用宿主机端口：

    • HTTP：9200
    • Transport：9300

  • 生产建议（需要你在系统层面自行设置）：

    • echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf ，执行命令 sudo sysctl -p 生效
    • 进程打开文件数 nofile >= 65536
    • 结合 bootstrap.memory_lock: true，确保 memlock 相关限制满足
    • 关闭 SWAP： sudo swapoff -a

创建持久化数据目录：

mkdir -p /opt/opensearch-cluster

mkdir -p /opt/opensearch-cluster/{config/opensearch/certs,data,logs}

mkdir -p /opt/opensearch-cluster/config/opensearch/opensearch-security

整体目录结构如下：

.
├── config
│   └── opensearch
│         ├── certs
│         │     ├── opensearch-ca.key
│         │     ├── opensearch-ca.p12
│         │     ├── opensearch-ca.pem
│         │     ├── instances.yml
│         │     ├── opensearch-1
│         │     │     └── opensearch-1.p12
│         │     ├── opensearch-2
│         │     │     └── opensearch-2.p12
│         │     └── opensearch-3
│         │           └── opensearch-3.p12
│         ├── jvm.options.d
│         ├── opensearch-security
│         │     ├── action_groups.yml
│         │     ├── allowlist.yml
│         │     ├── audit.yml
│         │     ├── config.yml
│         │     ├── internal_users.yml
│         │     ├── nodes_dn.yml
│         │     ├── roles_mapping.yml
│         │     ├── roles.yml
│         │     └── tenants.yml
│         └── opensearch.yml
├── data
├── docker-compose.yaml
├── logs
└── README.md

通用的 Docker Compose 配置

docker-compose.yaml
services:
  opensearch:
    image: opensearchproject/opensearch:3.5.0
    container_name: opensearch
    restart: unless-stopped
    
    network_mode: host      # 推荐使用 Host Network 模式，否则网络很难管理，容易出问题

    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536

    environment:
      - OPENSEARCH_JAVA_OPTS=-Xms16g -Xmx16g
      - DISABLE_INSTALL_DEMO_CONFIG=true
      - TZ=Asia/Shanghai

    volumes:
      - ./config/opensearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml:ro
      - ./config/opensearch/jvm.options.d:/usr/share/opensearch/config/jvm.options.d:ro
      - ./config/opensearch/certs:/usr/share/opensearch/config/certs:ro
      - ./config/opensearch/opensearch-security:/usr/share/opensearch/config/opensearch-security:ro
      - ./data:/usr/share/opensearch/data

Opensearch 主配置文件 ./config/opensearch/opensearch.yml ，以第一个节点为例：

此文件在每台节点机器上都需要 同结构、不同节点差异 的一份（通常做法：三台机器各自拷贝本目录，然后分别修改本机的 node.name、network.publish_host、以及 TLS keystore 文件路径）。

./config/opensearch/opensearch.yml
cluster.name: opensearch-prod
node.name: opensearch-1

network.host: 0.0.0.0
network.publish_host: 172.16.10.72

http.port: 9200
transport.port: 9300

bootstrap.memory_lock: true

node.roles:
  - cluster_manager
  - data
  - ingest

discovery.seed_hosts:
  - 172.16.10.72:9300
  - 172.16.10.70:9300
  - 172.16.10.71:9300

cluster.initial_cluster_manager_nodes:
  - opensearch-1
  - opensearch-2
  - opensearch-3

plugins.security.disabled: false

# =========================
# transport layer TLS
# =========================
plugins.security.ssl.transport.keystore_type: PKCS12
plugins.security.ssl.transport.keystore_filepath: certs/opensearch-1/opensearch-1.p12
plugins.security.ssl.transport.keystore_password: "changeit"
plugins.security.ssl.transport.keystore_keypassword: "changeit"

plugins.security.ssl.transport.truststore_type: PKCS12
plugins.security.ssl.transport.truststore_filepath: certs/opensearch-ca.p12
plugins.security.ssl.transport.truststore_password: "changeit"

# =========================
# REST layer TLS
# =========================
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.keystore_type: PKCS12
plugins.security.ssl.http.keystore_filepath: certs/opensearch-1/opensearch-1.p12
plugins.security.ssl.http.keystore_password: "changeit"
plugins.security.ssl.http.keystore_keypassword: "changeit"

plugins.security.ssl.http.truststore_type: PKCS12
plugins.security.ssl.http.truststore_filepath: certs/opensearch-ca.p12
plugins.security.ssl.http.truststore_password: "changeit"

# 通常保持 NONE/OPTIONAL，除非你要双向认证
plugins.security.ssl.http.clientauth_mode: OPTIONAL

# =========================
# node certificate identity
# =========================
plugins.security.nodes_dn:
  - "CN=opensearch-1"
  - "CN=opensearch-2"
  - "CN=opensearch-3"

plugins.security.authcz.admin_dn:     # 必须和节点证书独立
  - "CN=opensearch-admin"

plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: false

plugins.security.restapi.roles_enabled:
  - "all_access"
  - "security_rest_api_access"


  • plugins.security.nodes_dn 需要覆盖集群内所有节点证书的 DN（本文档以 CN=opensearch-{1,2,3} 为例）。
  • plugins.security.authcz.admin_dn 指定可以执行安全管理动作（如初始化安全索引）的 admin 证书 DN。这里示例为 CN=opensearch-1。
  • changeit 仅为示例密码；生产建议改为强密码，并用 opensearch.keystore 的 *_secure 配置项替代明文密码。

Opensearch Security 配置文件 config/opensearch/opensearch-security/ ，里面需要有 安全索引初始化 所需的安全相关配置文件（无需所有节点都存在，只需要在执行安全索引初始化的节点上存在这些配置即可）：

  • config.yml

    config.yml
    _meta:
      type: "config"
      config_version: 2

    config:
      dynamic:
        http:
          anonymous_auth_enabled: false
          xff:
            enabled: false
            internalProxies: ".*"
            remoteIpHeader: "X-Forwarded-For"
        authc:
          basic_internal_auth_domain:
            http_enabled: true
            transport_enabled: true
            order: 0
            http_authenticator:
              type: "basic"
              challenge: true
            authentication_backend:
              type: "internal"

  • action_groups.yml

    action_groups.yml
    _meta:
      type: "actiongroups"
      config_version: 2

    all_access:
      reserved: true
      allowed_actions:
        - "*"

    security_rest_api_access:
      reserved: true
      allowed_actions:
        - "cluster:admin/opensearch/security/*"

  • allowlist.yml

    allowlist.yml
    _meta:
      type: "allowlist"
      config_version: 2

    allowlist:
      enabled: false
      requests: {}

  • audit.yml

    audit.yml
    _meta:
      type: "audit"
      config_version: 2

    config:
      enabled: false

  • internal_users.yml ，修改 admin 密码需要更新此文件，命重新运行安全索引初始化

    internal_users.yml
    _meta:
      type: "internalusers"
      config_version: 2

    admin:
      # password: xOKcu3FXOEpORTc67 这里需要存储 admin （超级管理员用户）明文密码加密后的 hash 值
      hash: "$2y$12$SbOF0VDs5bzzqvqaFkvTI.KAWailXH"
      reserved: true
      backend_roles:
        - "admin"
      description: "Admin user"

  • nodes_dn.yml 里面需要配置所有的节点证书的 DN，防止没有在里面的 DN 节点加入集群

    nodes_dn.yml
    _meta:
      type: "nodesdn"
      config_version: 2

    nodes_dn:
      nodes_dn:
        - "CN=opensearch-1"
        - "CN=opensearch-2"
        - "CN=opensearch-3"

  • roles_mapping.yml

    roles_mapping.yml
    _meta:
      type: "rolesmapping"
      config_version: 2

    all_access:
      reserved: true
      users:
        - "admin"
      backend_roles:
        - "admin"

    security_rest_api_access:
      reserved: true
      users:
        - "admin"
      backend_roles:
        - "admin"

  • roles.yml

    roles.yml
    _meta:
      type: "roles"
      config_version: 2

    all_access:
      reserved: true
      cluster_permissions:
        - "*"
      index_permissions:
        - index_patterns:
            - "*"
          allowed_actions:
            - "*"

    security_rest_api_access:
      reserved: true
      cluster_permissions:
        - "cluster:admin/opensearch/security/*"

  • tenants.yml

    tenants.yml
    _meta:
      type: "tenants"
      config_version: 2

    global_tenant:
      reserved: true
      description: "Global tenant"

接下来为集群生成所需的证书，在每一个节点上执行生成各自节点的证书（CA 证书共用一套 opensearch-ca.p12 与 opensearch-ca.pem）。包括

  • 一个 CA truststore （Java 可识别的 PKCS12，至少包含 1 个 trustedCertEntry）
  • 每个节点一个 PKCS12 keystore （含该节点私钥与证书）
  • 证书的 SAN（IP/DNS） 参考 config/opensearch/certs/instances.yml
  • 一个 opensearch-admin 安全索引初始化证书

  1. 生成 config/opensearch/certs/instances.yml 文件用于批量生成节点证书时作为参考，证书要包含以下 SAN（IP/DNS） 信息，不生成也可以

    config/opensearch/certs/instances.yml
    instances:
      - name: opensearch-1
        dns:
          - opensearch-1
          - localhost
        ip: 
          - 172.16.10.72
          - 127.0.0.1

      - name: opensearch-2
        ip: 
          - 172.16.10.70
          - 127.0.0.1
        dns:
          - opensearch-2
          - localhost

      - name: opensearch-3
        ip: 
          - 172.16.10.71
          - 127.0.0.1
        dns:
          - opensearch-3
          - localhost
       
      - name: opensearch-admin
        ip:
          - 127.0.0.1
        dns:
          - localhost
          - opensearch-admin



  2. 生成 CA（得到 opensearch-ca.pem 与 opensearch-ca.p12）

    cd /opt/opensearch-cluster
    umask 077
    PASS='changeit'
    CA_NAME='opensearch-ca'

    openssl genrsa -out "config/opensearch/certs/${CA_NAME}.key" 4096
    openssl req -x509 -new -nodes \
      -key "config/opensearch/certs/${CA_NAME}.key" \
      -sha256 -days 3650 \
      -subj "/CN=${CA_NAME}" \
      -out "config/opensearch/certs/${CA_NAME}.pem"

    docker run --rm \
      -v /opt/opensearch-cluster/config/opensearch/certs:/certs:rw \
      -e PASS="${PASS}" \
      -e CA_NAME="${CA_NAME}" \
      --entrypoint sh \
      opensearchproject/opensearch:3.5.0 \
      -lc 'set -e; rm -f "/certs/${CA_NAME}.p12"; /usr/share/opensearch/jdk/bin/keytool -importcert -noprompt -alias ca -file "/certs/${CA_NAME}.pem" -keystore "/certs/${CA_NAME}.p12" -storetype PKCS12 -storepass "$PASS"'

    chmod 600 "config/opensearch/certs/${CA_NAME}.key" "config/opensearch/certs/${CA_NAME}.pem" "config/opensearch/certs/${CA_NAME}.p12"


  3. 为每个节点生成证书并打包为 PKCS12，此处以 opensearch-1 节点为例，其他节点参考执行即可

    cd /opt/opensearch-cluster
    umask 077
    PASS='changeit'
    CA_NAME='opensearch-ca'
    NODE='opensearch-1'
    IP='172.16.10.72'
    DNS='opensearch-1'

    mkdir -m 700 -p "config/opensearch/certs/${NODE}"

    openssl genrsa -out "config/opensearch/certs/${NODE}/${NODE}.key" 2048

    cat >"config/opensearch/certs/${NODE}/${NODE}.csr.cnf" <<EOF
    [req]
    distinguished_name = dn
    req_extensions = v3_req
    prompt = no

    [dn]
    CN = ${NODE}

    [v3_req]
    keyUsage = critical, digitalSignature, keyEncipherment
    extendedKeyUsage = serverAuth, clientAuth
    subjectAltName = @alt_names

    [alt_names]
    DNS.1 = ${DNS}
    DNS.2 = localhost
    IP.1 = ${IP}
    IP.2 = 127.0.0.1
    EOF

    openssl req -new \
      -key "config/opensearch/certs/${NODE}/${NODE}.key" \
      -out "config/opensearch/certs/${NODE}/${NODE}.csr" \
      -config "config/opensearch/certs/${NODE}/${NODE}.csr.cnf"

    openssl x509 -req \
      -in "config/opensearch/certs/${NODE}/${NODE}.csr" \
      -CA "config/opensearch/certs/${CA_NAME}.pem" \
      -CAkey "config/opensearch/certs/${CA_NAME}.key" \
      -CAcreateserial \
      -out "config/opensearch/certs/${NODE}/${NODE}.crt" \
      -days 825 -sha256 \
      -extensions v3_req \
      -extfile "config/opensearch/certs/${NODE}/${NODE}.csr.cnf"

    openssl pkcs12 -export \
      -name "${NODE}" \
      -inkey "config/opensearch/certs/${NODE}/${NODE}.key" \
      -in "config/opensearch/certs/${NODE}/${NODE}.crt" \
      -certfile "config/opensearch/certs/${CA_NAME}.pem" \
      -out "config/opensearch/certs/${NODE}/${NODE}.p12" \
      -passout "pass:${PASS}"

    对 opensearch-2 / opensearch-3 重复上述步骤，替换 NODE/IP/DNS。

  4. 生成 admin 证书（ opensearch-admin ）（用于集群安全索引初始化）

    cd /opt/opensearch-cluster
    PASS='changeit'
    CA_NAME='opensearch-ca'
    ADMIN_DN='opensearch-admin'

    mkdir -p config/opensearch/certs/admin
    umask 077

    openssl genrsa -out config/opensearch/certs/admin/admin.key 2048

    cat > config/opensearch/certs/admin/admin.csr.cnf <<EOF
    [req]
    distinguished_name = dn
    req_extensions = v3_req
    prompt = no

    [dn]
    CN = ${ADMIN_DN}

    [v3_req]
    keyUsage = critical, digitalSignature, keyEncipherment
    extendedKeyUsage = clientAuth
    EOF

    openssl req -new \
      -key config/opensearch/certs/admin/admin.key \
      -out config/opensearch/certs/admin/admin.csr \
      -config config/opensearch/certs/admin/admin.csr.cnf

    openssl x509 -req \
      -in config/opensearch/certs/admin/admin.csr \
      -CA  config/opensearch/certs/${CA_NAME}.pem \
      -CAkey config/opensearch/certs/${CA_NAME}.key \
      -CAcreateserial \
      -out config/opensearch/certs/admin/admin.crt \
      -days 825 -sha256 \
      -extensions v3_req \
      -extfile config/opensearch/certs/admin/admin.csr.cnf

    openssl pkcs12 -export \
      -name "${ADMIN_DN}" \
      -inkey config/opensearch/certs/admin/admin.key \
      -in config/opensearch/certs/admin/admin.crt \
      -certfile config/opensearch/certs/${CA_NAME}.pem \
      -out config/opensearch/certs/admin/admin.p12 \
      -passout "pass:${PASS}"

    chmod 600 config/opensearch/certs/admin/admin.*

    把 config/opensearch/certs/admin/admin.p12 （以及 CA truststore opensearch-ca.p12 ）同步到三台节点相同路径（保证容器内都能读到）。

在所有节点上启动 Opensearch

docker compose up -d

安全索引初始化（必须执行一次）

当三台节点都启动并且集群选主成功后，需要初始化 OpenSearch Security 的配置索引 .opendistro_security。否则日志会持续提示：

no such index [.opendistro_security] retrieving configuration ...

初始化命令（在 opensearch-1 上执行一次即可；会写入数据目录 ./data），这里需要用到之前生成的 opensearch-admin 证书：

cd /opt/opensearch-cluster
PASS='changeit'

$ docker compose exec opensearch /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -icl -nhnv \
  -h 127.0.0.1 -p 9200 \
  -cd /usr/share/opensearch/config/opensearch-security \
  -ks /usr/share/opensearch/config/certs/admin/admin.p12 \
  -kst PKCS12 \
  -ksalias opensearch-admin \
  -kspass ${PASS} \
  -ts /usr/share/opensearch/config/certs/opensearch-ca.p12 \
  -tst PKCS12 \
  -tspass ${PASS}
  
Security Admin v7
Will connect to 127.0.0.1:9200 ... done
Connected as "CN=opensearch-admin"
OpenSearch Version: 3.5.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-prod
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/config/opensearch-security/
Will update '/config' with /usr/share/opensearch/config/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /usr/share/opensearch/config/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /usr/share/opensearch/config/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/audit' with /usr/share/opensearch/config/opensearch-security/audit.yml
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /usr/share/opensearch/config/opensearch-security/allowlist.yml
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 9 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"],"updated_config_size":9,"message":null} is 9 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","actiongroups","config","internalusers"]) due to: null
Done with success

config/opensearch/opensearch-security/internal_users.yml 中的 hash 值是 admin 用户的 hash 加密值，要修改密码, 可以使用 hash.sh 工具生成新的 hash 值，然后替换 internal_users.yml 中的 hash 字段。

docker compose exec opensearch \
  /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p "$ADMIN_PASS"

会生成指定密码的 Hash 值，将其更新到 internal_users.yml 中，执行安全索引初始化命令重新上传安全配置。

cd /opt/opensearch-cluster
PASS='changeit'

$ docker compose exec opensearch /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -icl -nhnv \
  -h 127.0.0.1 -p 9200 \
  -cd /usr/share/opensearch/config/opensearch-security \
  -ks /usr/share/opensearch/config/certs/admin/admin.p12 \
  -kst PKCS12 \
  -ksalias opensearch-admin \
  -kspass ${PASS} \
  -ts /usr/share/opensearch/config/certs/opensearch-ca.p12 \
  -tst PKCS12 \
  -tspass ${PASS}