tencent cloud

Feedback

Redis Instance Connection Failure

Last updated: 2023-06-06 15:38:52

    Issue

    Issue 1: Use a CVM instance to connect to a TencentDB for Redis instance at the private network address automatically assigned by the system as instructed in Connecting to TencentDB for Redis Instance, but the connection fails.
    Issue 2: Log in to the TencentDB for Redis console, go to the instance list, click Log In in the Operation column of the target instance to redirect to DMC, connect to the instance, but the connection fails.

    Possible Causes

    For database connection failures for the first time, the possible causes are as follows:
    Port error.
    Network configuration error or incorrect security group configuration.
    Password error.
    For sudden connection failures during instance running, the possible causes are as follows:
    The maximum number of connections has been reached.
    Memory or shards have been used up.
    A high-availability (HA) switch occurred, the database service became unavailable, a read-only replica switch occurred, or the read-only replica service became unavailable, etc.
    For client errors, the possible causes are as follows:
    Connection pool parameter setting is unreasonable.
    Connections leaked.

    Troubleshooting

    Step 1. Run telnet to check whether the Redis port can be accessed normally

    Run telnet in the command line tool to narrow down the cause of the error:
    [root@VM-4-10-centos ~]# telnet 10.x.x.34 6379
    Trying 10.x.x.34...
    Connected to 10.x.x.34.
    Escape character is '^]'.
    As shown above, successful connection means that the port of the Redis instance can be accessed normally. If there is an exception, go to Step 2 to troubleshoot network issues.

    Step 2. Check whether it is caused by network configuration

    To connect over the private network, the CVM and TencentDB instances must be under the same account and in the same VPC, or both in the classic network. The connection will fail in the following conditions:
    If the CVM instance is in a VPC, while the Redis instance in the classic network, we recommend that you switch the network type of the Redis instance from classic network to VPC.
    If the Redis instance is in a VPC, while the CVM instance in the classic network, we recommend that you switch the network type of the CVM instance from classic network to VPC. For more information, see Switching to VPC.
    If the CVM and TencentDB for Redis instances are in different VPCs in the same region, we recommend that you migrate the Redis instance to the VPC of the CVM instance.
    If the CVM and TencentDB for Redis instances are in different VPCs in different regions, we recommend that you create a CCN between the two VPCs.
    If the CVM and TencentDB for Redis instances are in different VPCs under different accounts, we recommend that you create a CCN between the two VPCs.

    Step 3. Troubleshoot the security group

    The CVM instance cannot connect to the TencentDB for Redis instance if their security groups are incorrect.

    Incorrect CVM security group configuration

    To use the CVM instance to access the Redis instance, you need to configure an outbound rule in the security group of the CVM instance. If the target of the outbound rule isn't "0.0.0.0/0" and the protocol port isn't "ALL", the IP and port of the Redis instance should be added to the rule.
    1. Go to the Security Group page in the CVM console and click the name of the CVM-bound security group to enter its details page.
    2. On the Outbound rule tab, click Add Rule.
    Type: Select Custom.
    Target: Enter the IP or IP range of your Redis instance.
    Protocol Port: Enter the private network port of the Redis instance.
    Policy: Select Allow.

    Incorrect Redis security group configuration

    To use the CVM instance to access the Redis instance, you need to configure an inbound rule in the security group of the Redis instance. If the source of the inbound rule isn't "0.0.0.0/0" and the protocol port isn't "ALL", the IP and port of the CVM instance should be added to the rule.
    1. Go to the Security Group page in the CVM console and click the name of the Redis-bound security group to enter its details page.
    2. On the Inbound rule tab, click Add Rule. Note that you also need to open the IP and port of the Redis instance in the inbound rule.
    Type: Select Custom.
    Source: Enter the IP or IP range of your CVM instance.
    Protocol Port: Enter the private network port of the Redis instance.
    Policy: Select Allow.
    Note
    The Redis instance uses private network port 6379 by default and supports customizing its port. If the default port is changed, the new port should be opened in the inbound rule of the Redis security group.
    If the default port 6379 of the Redis instance is used, it should be opened in the inbound rule of the Redis security group.

    Step 4. Troubleshooting the password

    Run the info command. If the following information is displayed, the password of the TencentDB for Redis instance is correct.
    [root@SNG-Qcloud /data/home/rickyu]# redis-cli -h 10.x.x.34 -p 6379 -a password
    10.x.x.2:6379> info cpu
    # CPU
    used_cpu_sys:1623.176000
    used_cpu_user:4649.572000
    used_cpu_sys_children:0.000000
    used_cpu_user_children:0.000000
    If NOAUTH Authentication required. is displayed, the password is incorrect.
    10.x.x.31:6379> info memory
    NOAUTH Authentication required.
    10.x.x.31:6379>
    Log in to the TencentDB for Redis console and click an instance ID in the instance list to enter the instance details page, where you can reset the password. For more information, see Managing Account.

    Step 5. Check whether the memory or shards are used up

    If the business reports the following error message:
    "-READONLY You can't write against a read only slave.\\r\\n"
    Log in to the TencentDB for Redis console, find the target instance in the instance list, click the instance ID to enter the System Monitoring page, and select Memory Utilization in the Metric drop-down list to view the memory utilization of the instance.
    If memory is used up, writes will fail. In this case, perform the following operations:
    Expand the capacity immediately as instructed in Changing Instance Specification. 
    Modify the database eviction policy as instructed in Managing Instance Parameter. In Parameter Settings, set the maxmemory-policy parameter to allkeys-lru or volatile-lru.
    Note
    Instance data may be lost if the allkeys-lru eviction policy is adopted. Assess the impact before doing so.

    Step 6. Check the connection quota

    The Connection Utilization metric refers to the ratio of the number of TCP connections from the client to the instance to the maximum number of connections to the instance. If this metric stays high, the current database connection quota is insufficient, and the maximum number of connections needs to be adjusted.

    Issue

    The error message is as follows:
    ERR max number of clients reached

    Solution

    1. Log in to the TencentDB for Redis console, Above the instance list on the right, select the region. In the instance list, find the target instance. Click the instance ID in blue to enter the Instance Details page, click the System Monitoring tab, and then select the Monitoring Metrics tab to view the monitoring data. In the View drop-down list, select Instance Monitoring and Connection Utilization as the metric, and check whether it stays high in the monitoring view.
    2. If the connection utilization stays high, adjust the maximum number of connections and modify the specifications for the number of connections. For details, see Adjusting the Number of Connections.

    Step 7. Check whether any of the following occurred: HA switch, unavailable database service, read-only replica switch, or unavailable read-only replica service

    If you find abnormal connections or a large number of access errors and slow queries at a certain point in time, and you also receive TCOP event alarms for those abnormal events, contact us for help.
    For the configuration method of TCOP event alarms, see Creating Event Rule

    Step 8. Confirm whether the configuration of the Jedis connection pool is correct if you use it

    Issue

    If the number of available connections in the connection pool is used up and the old connections are not released in time, the newly created connection will fail, and the client will prompt the following error message.
    JedisConnectionException: Could not get a resource from the pool

    Solution

    1. Use the following command on the client to confirm the number of connections currently accessing port 6379 of the instance. If this number is close to the maxTotal value configured in the connection pool, a connection failure will occur.
    netstat -an | grep 6379 | grep ESTABLISHED | wc -l
    2. See Java Connection Sample, and check whether to call jedis.close() to release old connections to avoid connection leaks.
    3. If all old connections have been released and the concurrent business volume is large, the maxTotal parameter value needs to be increased.
    Note
    The maxTotal value of each client connection pool * the number of clients = the maximum number of connections for TencentDB for Redis.
    

    References

    Viewing the network type and VPC information

    To enable connection between CVM and TencentDB for Redis instances over private network, they must be under the same account and in the same VPC, or both in the classic network.
    Note
    If the Network fields in the instance lists both show Classic Network or VPC, it means that the networks of the CVM and TencentDB for Redis instances are of the same type.
    If the Network fields in the instance lists both show the same VPC (in the same region), it means that the CVM and TencentDB for Redis instances are in the same VPC.

    Viewing the CVM network type

    Log in to the CVM console and view the network information in the instance list.

    Viewing the Redis network type

    Log in to the TencentDB for Redis console and view the network information in the instance list.
    

    Enabling public network access

    TencentDB for Redis now allows you to manually enable public network access in the console, so that instances can be accessed over public network. For detailed directions, see Configuring the Public Network Address.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support