tencent cloud

• UDP listeners cannot get the real client IP through TOA.
• If there are already TOA-related operations between the client and the real server, the real server may fail to obtain the real client IP.
• After TOA is inserted, it takes effect only on new connections.
• TOA may degrade the server performance as it needs to perform additional processing such as extracting the address from the field TCP option.
• Compatibility issues may come up when Tencent Cloud TOA is used with other TOA modules.
• Tencent Cloud TOA is embedded in TencentOS and can be used to obtain the real source IP in hybrid cloud deployment scenarios. If the server is run on TencentOS and deployed in a hybrid cloud, you can directly execute the command modprobe toa to load TOA. If the server is run on Linux, Linux TOA should be used instead.

To get the real source IP in the NAT64 CLB scenario, you need to modify the source code after the toa.ko kernel module is inserted into the real server and TOA will pass the IP address to the real server.

1. Define a data structure to store the IP address.
   

   struct toa_nat64_peer {  
   struct in6_addr saddr;  
   uint16_t sport;  
   };  
   ....  
   struct toa_nat64_peer client_addr;  
   ....  
   

2. Define TOA variables and make calls to get the real source IPv6 address.
   

   enum {  
   TOA_BASE_CTL            = 4096,  
   TOA_SO_SET_MAX          = TOA_BASE_CTL,  
   TOA_SO_GET_LOOKUP       = TOA_BASE_CTL,  
   TOA_SO_GET_MAX          = TOA_SO_GET_LOOKUP,  
   };  
   getsockopt(client_fd, IPPROTO_IP, TOA_SO_GET_LOOKUP, &client_addr, &len);  
   

3. Obtain the source IP address.
   

   real_ipv6_saddr = client_addr.saddr;  
   real_ipv6_sport = client_addr.sport;
   

A complete configuration sample is as follows:

//Define TOA variables. Set `TOA_BASE_CTL` to 4096.
enum {  
TOA_BASE_CTL            = 4096,  
TOA_SO_SET_MAX          = TOA_BASE_CTL,  
TOA_SO_GET_LOOKUP       = TOA_BASE_CTL,  
TOA_SO_GET_MAX          = TOA_SO_GET_LOOKUP,  
};  
//Define a data structure to store the IP address.
struct toa_nat64_peer {  
struct in6_addr saddr;  
uint16_t sport;  
};  
//Declare the variable that is used to store the address.  
struct toa_nat64_peer client_addr;  
.……  
//Get the file descriptor of the client, where `listenfd` is the listening file descriptor of the server.  
client_fd = accept(listenfd, (struct sockaddr*)&caddr, &length);  
//Make calls to get the real source IP of the user in the NAT64 scenario.  
char from[40];  
int len = sizeof(struct toa_nat64_peer);  
if (getsockopt(client_fd, IPPROTO_IP, TOA_SO_GET_LOOKUP, &client_addr, &len) == 0) {  
inet_ntop(AF_INET6, &client_addr.saddr, from, sizeof(from));  
//Obtain the source IP and source port  
printf("real client [%s]:%d\n", from, ntohs(client_addr.sport));  
}  

To get the real source IP in the scenario where NAT64 CLB is used in a hybrid cloud, you need to modify the source code after the toa.ko kernel module is inserted into the real server and TOA will pass the IP address to the real server.

A complete configuration sample is as follows:

//Define TOA variables. Set `TOA_BASE_CTL` to 4096.  
enum {  
TOA_BASE_CTL = 4096,  
TOA_SO_SET_MAX = TOA_BASE_CTL,  
TOA_SO_GET_LOOKUP = TOA_BASE_CTL,  
TOA_SO_GET_MAX = TOA_SO_GET_LOOKUP,   
};  
//Define a data structure to store the IP address.    
struct toa_nat64_peer {    
struct in6_addr saddr;    
uint16_t sport;    
};    
//Declare the variable that is used to store the address.    
struct toa_nat64_peer client_addr_nat64;    
.......  
//Get the file descriptor of the client, where `listenfd` is the listening file descriptor of the server.  
//Make calls to get the real client IP in the NAT64 scenario.   
char from[40];    
int len = sizeof(struct toa_nat64_peer);  
int ret;  
ret = getsockopt(client_fd, IPPROTO_IP, TOA_SO_GET_LOOKUP, &client_addr_nat64, &len);  
if (ret == 0) {  
inet_ntop(AF_INET6, &(client_addr_nat64.saddr), from, sizeof(from));    
//Obtain the source IP and source port.   
printf("real client v6 [%s]:%d\n", from, ntohs(client_addr_nat64.sport));   
} else if (ret != 0) {  
struct sockaddr v4addr;  
len = sizeof(struct sockaddr);  
//Obtain the source IP and source port:  
//In the hybrid cloud deployment scenario, the source IP address is the IP address after SNAT;  
//In the non-hybrid cloud deployment scenario, the source IP address is the client IP address without SNAT and NAT64.  
//The semantics of this function is to get the real client address and port.  
if (get_peer_name(client_fd, &v4addr, &len) == 0) {  
    inet_ntop(AF_INET, &(((struct sockaddr_in *)&v4addr)->sin_addr), from, sizeof(from));  
    printf("real client v4 [%s]:%d\n", from, ntohs(((struct sockaddr_in *)&v4addr)->sin_port));   
}  
}  

Run the following command to check the TOA metrics.

cat /proc/net/toa_stats

• Run the following command to check my kernel version. If you see tlinux in the command output, you are using TLinux OS, while linux indicates you are using Linux OS.
  

  uname -a
  

  
  

• You can also check the version using the following command. If tlinux or tlinux is returned, you are using TLinux OS.
  

  rpm -qa | grep kernel
  

  

1. Run the following command to check whether TOA has been loaded.
   

   lsmod | grep toa
   

   
2. Check whether the server program has made correct calls to get the source IP. You can refer to Adapt the Real Sever.
3. Capture TCP packets on the server and check whether the packets contain the source IP information.

• if unknown-200 is displayed in the tcp option output, it indicates that the real source IP after SNAT is inserted into the field tcp option.
• If unknown-253 is displayed, it indicates that the real source IPv6 address is inserted in the NAT64 scenario.
  

4. If the packets containing the TOA address has been sent to the server, compile toa.ko to a DEBUG version, and further locate the problem through the kernel log. In the downloaded TOA source directory, add the DEBUG compilation option to the make file.
5. Run the following command to compile again.
   

   make clean
   make
   

6. Run the following commands to uninstall the original toa.ko and install the latest one.
   

   rmmod toa 
   insmod ./toa.ko
   

7. Run the following command to observe the kernel log.
   

   dmesg -Tw
   

   
   If you see the following message, TOA is working normally. You can further check whether the server program has made calls to get the real source IP, or whether the API is used incorrectly.
   
8. If you failed to find out the problem with the preceding steps, submit a ticket.