Configure MongoDB Cluster for TLS/SSL

Configure MongoDB Cluster for TLS/SSL

ReGY 2,907 2021-06-22

Starting in version 4.0, MongoDB disables support for TLS 1.0 encryption on systems where TLS 1.1+ is available.

Certificate Requirement

  • any certificate needs to be signed by the same CA
  • the common name (CN) required during the certificate creation must correspond to the hostname of the host
  • any other field requested in the certificate creation should be a non-empty value and, hopefully, should reflect our organization details
  • it is also very important that all the fields, except the CN, should match those from the certificates for the other cluster members

Configure Internal X.509 certificate-based Encryption

Configure example-ca.conf

# For the CA policy
[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]
default_bits = 4096
default_keyfile = example-ca.pem    ## The default private key file name.
default_md = sha256                           ## Use SHA-256 for Signatures
distinguished_name = req_dn
req_extensions = v3_req
x509_extensions = v3_ca # The extentions to add to the self signed cert

[ v3_req ]
subjectKeyIdentifier  = hash
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
nsComment = "OpenSSL Generated Certificate for TESTING only.  NOT FOR PRODUCTION USE."
extendedKeyUsage  = serverAuth, clientAuth

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = TW

countryName_min = 2
countryName_max = 2

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = TW
stateOrProvinceName_max = 64

localityName = Locality Name (eg, city)
localityName_default = Taipei
localityName_max = 64

organizationName = Organization Name (eg, company)
organizationName_default = Omniwaresoft
organizationName_max = 64

organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = no-sql
organizationalUnitName_max = 64

commonName = Common Name (eg, YOUR name)
commonName_max = 64

[ v3_ca ]
# Extensions for a typical CA

subjectKeyIdentifier=hash
basicConstraints = critical,CA:true
authorityKeyIdentifier=keyid:always,issuer:always

Connect to one of the hosts and generate a new private key using openssl

# created a new 8192-bit private key and saved it in the file example-ca.key
$ openssl genrsa -out example-ca.key -aes256 8192

Sign a new CA certificate

# Creating certificate and filling out some fields correspond to our org's details
$ openssl req -x509 -new -extensions v3_ca -key mongoCA.key -days 365 -out mongoCA.crt

#####################################################################################################################################
# v0.0.2
[root@mongodb-regy-4 tls]# openssl req -x509 -new -extensions v3_ca -key example-ca.key -days 365 -out example-ca-pub.crt -config example-ca.conf
Enter pass phrase for example-ca.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:TW
State or Province Name (full name) []:TW
Locality Name (eg, city) [Default City]:Taipei
Organization Name (eg, company) [Default Company Ltd]:Omniwaresoft
Organizational Unit Name (eg, section) []:no-sql
Common Name (eg, your name or your server's hostname) []:mongodb-regy-4.c.omnilabs-1281.internal
Email Address []:[email protected]

Configure mongod_cert.conf

The most important part is [ alt_names ]. As explained before, the DNS must include include both internal endpoints and external endpoints:

# Could check your Ops Manager Host Mappings as well

[ alt_names ]
DNS.1 = mongodb-regy-4.c.omnilabs-1281.internal
DNS.2 = mongodb-regy-4

mongod_cert.conf:

[ req ]
default_bits = 4096
default_md = sha256
distinguished_name = req_dn
req_extensions = v3_req

[ v3_req ]
subjectKeyIdentifier  = hash
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
nsComment = "OpenSSL Generated Certificate for TESTING only.  NOT FOR PRODUCTION USE."
extendedKeyUsage  = serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 =         ##TODO: Enter the DNS names. The DNS names should match the server names.
DNS.2 =         ##TODO: Enter the DNS names. The DNS names should match the server names.
IP.1 =          ##TODO: Enter the IP address. SAN matching by IP address is available starting in MongoDB 4.2
IP.2 =          ##TODO: Enter the IP address. SAN matching by IP address is available starting in MongoDB 4.2

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = TW

countryName_min = 2
countryName_max = 2

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = TW
stateOrProvinceName_max = 64

localityName = Locality Name (eg, city)
localityName_default = Taipei
localityName_max = 64

organizationName = Organization Name (eg, company)
organizationName_default = Omniwaresoft
organizationName_max = 64

organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = no-sql
organizationalUnitName_max = 64

commonName = Common Name (eg, YOUR name)
commonName_max = 64

Issue self-signed certificates for all the nodes

fill out all the fields requested the same for each host, but remember to fill out a different common name (CN) that must correspond to the hostname.

# generate a certificate request and sign it using the CA certificate
$ openssl req -new -nodes -newkey rsa:4096 -keyout mongotls1.key -out mongotls1.csr

$ openssl x509 -CA mongoCA.crt -CAkey mongoCA.key -CAcreateserial -req -days 365 -in mongotls1.csr -out mongotls1.crt

$ cat mongotls1.key mongotls1.crt > mongotls1.pem

# apply the same thing for the second and third nodes

#####################################################################################################################################
# v0.0.2

# generate a certificate request and sign it using the CA certificate
# generate sever key & csr
openssl genrsa -aes256 -passout pass:regy0415 -out mongod.key 2048
openssl req -new -key mongod.key -out mongod.csr -config mongod_cert.conf

# signed by the Root CA
openssl x509 -req -days 3650 -in mongod.csr -CA example-ca-pub.crt -CAkey example-ca.key -passin pass:regy0415@ -CAcreateserial -out mongod.crt -extfile mongod_cert.conf -extensions v3_req

# concate the key & crt file to pem
cat mongod.csr mongod.key > mongod.pem

# Verify key by CA
openssl verify -CAfile example-ca-pub.crt mongod.pem

# Repeat the above process and replace mongod

Verify Certificates

# Verify Server Certificate
openssl verify -CAfile example-ca-pub.crt mongod.pem
mongod.pem: OK

# Verify CA itself:
openssl verify -CAfile example-ca-pub.crt example-ca-pub.crt
example-ca-pub.crt: OK

View Certificate Detail

# check altnames in certificate
openssl x509 -text -noout -in mongod.pem
...
	X509v3 Subject Alternative Name:
			DNS:mongodb-regy-4.c.omnilabs-1281.internal, DNS:mongodb-regy-4, IP Address:xx.xx.xx.xx, IP Address:xx.xx.xx.xx
...

Place the files

# Create on each member a directory that only the MongoDB user can read, and copy both files there
$ sudo mkdir -p /etc/mongodb/tls
$ sudo chmod 600 /etc/mongodb/tls
$ sudo chown -R mongod:mongod /etc/mongodb

# Copy each self signed certifcate **<hostname>.pem** into the relative member
$ sudo cp mongod.pem /etc/mongodb/tls

# Copy to each node the CA certifcate file
$ sudo cp example-ca-pub.crt /etc/mongodb/tls

Configure mongod

# Change the configuration file /etc/mongod.conf on each host adding the following rows
net:
    tls:
       mode: requireTLS
       PEMKeyFile: /etc/mongodb/tls/mongod.pem
       CAFile: /etc/mongodb/tls/example-ca-pub.crt

# Restart Mongod Daemon
$ systemctl restart mongod

# checking configured replica set using encrypted connections
$ mongo --authenticationDatabase "admin" --host mongod:27017 --tls --tlsCAFile /etc/tls/example-ca-pub.crt --tlsPEMKeyFile /etc/mongodb/tls/mongod.pem -u dbuser -p passw0rd1

Configure TLS Connections to Ops Manager

Configure Ops Manager Application for TLS

  • Ops Manager Config
    • URL To Access Ops Manger

      including port 8443

    • HTTPS PEM Key File

      Private key & certificate(.pem file) for Ops Manager Host

    • Client Certificate Mode

      Select if client applications or MongoDB Agents must present a TLS certificate when connecting to a TLS-enabled Ops Manager. Ops Manager checks for certificates from these client hosts when they try to connect. If you choose to require the client TLS certificates, make sure they are valid.

      Accepted values are:

      • None
      • Required for Agents Only
      • Required for All Requests

Configure MongoDB Agent to Use TLS

  • Open MongoDB Agent configuration file

    vim /etc/mongodb-mms/automation-agent.config
    
  • Set or add the following properties where needed

    • mmsBaseUrl=https://example.opsm.com:8443

    • tlsRequireValidMMSServerCertificates

      The agent to validate TLS certificates of Ops Manager
      < - If you set this value to true, you must set httpsCAFile - >

    • httpsCAFile

      This Certificate Authority file must be in the same location on each MongoDB host in the same sharded cluster or replica set. Any MongoDB host that does not have the file in the same file location as the others may become unaccessible.

    • tlsMMSServerClientCertificate

      certificate of automation-agent

    • tlsMMSServerClientCertificatePassword

      Use the tlsMMSServerClientCertificatePasswordExec option instead of tlsMMSServerClientCertificatePassword to specify the password from a shell command.

    • automation-agent.config

      # Hostname of the Ops Manager web server. The hostname will match what is used
      # to access the Ops Manager UI. The default port for an Ops Manager install
      # is 8080.
      #
      # ex. http://opsmanaager.<company>.com:8080
      mmsBaseUrl=https://mongom-regy.c.omnilabs-1281.internal:8443
      
      # optional
      tlsRequireValidMMSServerCertificates=true
      
      httpsCAFile=/mongo/tls/example-ca-pub.crt
      
      tlsMMSServerClientCertificate=/mongo/tls/automation-agent.pem
      
      tlsMMSServerClientCertificatePassword=xxxxxxx
      
  • Restart automation-agent

    systemctl restart mongodb-mms-automation-agent
    

Enable TLS for a Deployment

With the Client Certificate Mode setting, you can set if the client must present a TLS certificate to connect to the deployments in your project. If you enable TLS for your project, all deployment must use TLS.

Enable TLS for the project

  • Security Tab
  • Authentication & TLS Settings
  • EDIT SETTINGS
  • (Authentication Mechanisms)
  • Specify the TLS Settings
    • Enable TLS
    • TLS CA File Path
    • Client Certificate Mode
      • OPTIONAL

        Every client may present a valid TLS certificate when connecting to MongoDB deployments. MongoDB Agents might use TLS certificates if you don’t set the mongod tlsMode to None.

      • REQUIRED

        Every MongoDB deployment in this project starts with TLS-encrypted network connections. All Agents must use TLS to connect to any MongoDB deployment.

Set Existing Deployments to Use TLS

If the project have not been enabled that will get an alert as below:

TLS%20SSL%20Configuration%20fa529573882c41bf92e5bbd582352ec7/Untitled.png

  • Modify Processes
  • Expand the Advanced Configuration Options section
  • Set the TLS/SSL startup options
    • tlsMode
    • tlsCertificateKeyFile
    • tlsCertificateKeyFilePassword
    • tlsFIPSMode(Optional)

OTHER

Enable x.509 Authentication

Credential Encrypted

MongoDB Driver TLS Connection String

FAQ

Should use the correct SAN(subject alternative name) to establish connection over TLS

  • Error between automation-agent and Ops Manager

    [root@mongodb-regy-4 mongodb-mms]# curl  https://10.140.0.3:8443
    curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
    [root@mongodb-regy-4 mongodb-mms]# curl -v  https://10.140.0.3:8443
    * About to connect() to 10.140.0.3 port 8443 (#0)
    *   Trying 10.140.0.3...
    * Connected to 10.140.0.3 (10.140.0.3) port 8443 (#0)
    * Initializing NSS with certpath: sql:/etc/pki/nssdb
    *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
      CApath: none
    * Server certificate:
    *       subject: [email protected],CN=mongom-regy.c.omnilabs-1281.internal,OU=no-sql,O=Omniwaresoft,L=Taipei,ST=TW,C=TW
    *       start date: May 17 08:47:46 2021 GMT
    *       expire date: Jun 16 08:47:46 2021 GMT
    *       common name: mongom-regy.c.omnilabs-1281.internal
    *       issuer: [email protected],CN=mongodb-regy-4.c.omnilabs-1281.internal,OU=no-sql,O=Omniwaresoft,L=Taipei,ST=TW,C=TW
    * NSS error -12276 (SSL_ERROR_BAD_CERT_DOMAIN)
    * Unable to communicate securely with peer: requested domain name does not match the server's certificate.
    * Closing connection 0
    curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
    
  • Error between automation-agent and MongoDB instances - mongodb.log

    2021-05-20T07:59:42.178+0000 I  CONNPOOL [Replication] Connecting to mongodb-regy-4.c.omnilabs-1281.internal:27021
    2021-05-20T07:59:42.184+0000 E  NETWORK  [Replication] SSL peer certificate validation failed: self signed certificate
    2021-05-20T07:59:42.188+0000 E  NETWORK  [Replication] SSL peer certificate validation failed: self signed certificate
    2021-05-20T07:59:42.194+0000 E  NETWORK  [Replication] SSL peer certificate validation failed: self signed certificate
    2021-05-20T07:59:42.194+0000 I  REPL_HB  [replexec-2] Heartbeat to mongodb-regy-4.c.omnilabs-1281.internal:27021 failed after 2 retries, response status: HostUnreachable: Error connecting to mongodb-regy-4.c.omnilabs-1281.internal:27021 (10.140.0.12:27021) :: caused by :: SSL peer certificate validation failed: self signed certificate
    2021-05-20T07:59:42.234+0000 I  NETWORK  [listener] connection accepted from 10.140.0.12:60388 #2063 (4 connections now open)
    2021-05-20T07:59:42.238+0000 E  NETWORK  [conn2063] SSL peer certificate validation failed: self signed certificate
    2021-05-20T07:59:42.238+0000 I  NETWORK  [conn2063] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate. Ending connection from 10.140.0.12:60388 (connection id: 2063)
    
  • Error between automation-agent and MongoDB instances - automation-agent.log

    [2021/05/20 12:30:14.170] [.error] [cm/mongoclientservice/mongoclientservice.go:sendClientToRequesters:609] [12:30:14.170] Error getting client ready for conn params = mongodb-regy-4.c.omnilabs-1281.internal:27020 (local=false).  Informing all requests and disposing of client (0x0).  requests=[ 0xc000db5500 ] : [12:30:14.170] Error dialing to connParams=mongodb-regy-4.c.omnilabs-1281.internal:27020 (local=false): tried 3 identities, but none of them worked.  They were (__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][1024], regy@$external[[MONGODB-X509]][0], )
    [2021/05/20 12:30:14.170] [.error] [cm/mongoclientservice/mongoclientservice.go:withClientForHelper:403] <hardwareMetricsCollector> [12:30:14.170] Error checking out client (0x0) for connParam=mongodb-regy-4.c.omnilabs-1281.internal:27020 (local=false) connectMode=SingleConnect : [12:30:14.170] Error dialing to connParams=mongodb-regy-4.c.omnilabs-1281.internal:27020 (local=false): tried 3 identities, but none of them worked.  They were (__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][1024], regy@$external[[MONGODB-X509]][0], )
    

# MongoDB # TLS/SSL # Security # X.509 # Ops Manager