tencent cloud

Feedback

Obtaining Real TCP Client IPs via TOA

Last updated: 2023-09-11 17:40:22
    You can use this document to learn how to get the TCP client IP via TOA when using L4 proxy.

    Use Cases

    Using L4 acceleration for data packets will have the source IP and port modified, so the origin does not get the original information. To enable the origin to get the real client IP and port, you can pass the information using TOA when creating an acceleration channel. In this way, the real client IP and port are passed into the TCP option field. Meanwhile, you need to install TOA on the origin to get that information.

    Directions

    Step 1: Pass the client IP via TOA

    To get the TCP client IP via TOA, set Pass client IP to TOA in L4 proxy forwarding rules in the console. For details on how to modify rules, see: Modifying an L4 Proxy Instance.
    
    
    

    Step 2: Load TOA on backend server

    You can load the TOA module using either of the following methods:
    Method 1 (recommended): Based on the Linux version the origin uses, download the complied toa.ko file and load it directly.
    Method 2: If you cannot find the appropriate Linux version, download the TOA source code file and compile and load it yourself.
    Note:
    Due to the differences in installation environments, if you encounter issues during the loading process using Method 1, please try Method 2 and install the compilation environment manually.
    Method 1: Download and load the compiled TOA module
    Method 2: Compile and load the TOA module
    1. Download and decompress the TOA package corresponding to the version of Linux OS on Tencent Cloud.
    centos
    debian
    suse linux
    ubuntu
    2. After decompression is complete, run the cd command to access the decompressed folder. Then load the module as follows:
    Load TOA with a script
    Manually load TOA
    /bin/bash -c "$(curl -fsSL https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/install_toa.sh)"
    When it is loaded successfully, you will see the following information:
    
    
    # Decompress the tar package.
    tar -zxvf CentOS-7.2-x86_64.tar.gz
    # Enter the directory of the decompressed package.
    cd CentOS-7.2-x86_64
    # Load the TOA module.
    insmod toa.ko
    # Copy the TOA module to the kernel module directory.
    cp toa.ko /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko
    # Configure the TOA module to load automatically at system startup.
    echo "insmod /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko" >> /etc/rc.local
    Run the following command to check whether the loading is successful:
    lsmod | grep toa
    If you see "TOA" in the message, the module is loaded successfully:
    
    
    
    
    1. Install the compilation environment.
    1.1 Make sure kernel-devel and kernel-headers are installed and consistent with the kernel version.
    1.2 Make sure the gcc and make dependencies are installed.
    1.3 If these environmental dependencies are not installed, run the installation command:
    CentOS
    Ubuntu/Debian
    yum install -y gcc
    yum install -y make
    yum install -y kernel-headers kernel-devel
    apt-get install -y gcc
    apt-get install -y make
    apt-get install -y linux-headers-$(uname -r)
    
    2. After the compilation environment is installed, download, compile and load the source code.
    Compile and load TOA with a script
    Manually compile and load TOA
    /bin/bash -c "$(curl -fsSL https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/compile_install_toa.sh)"
    # Create a compilation directory and enter it.
    mkdir toa_compile && cd toa_compile
    # Download the source code (tar.gz)
    curl -o toa.tar.gz https://edgeone-document-file-1258344699.cos.ap-guangzhou.myqcloud.com/TOA/toa.tar.gz
    # Decompress the tar package
    tar -zxvf toa.tar.gz
    # Compile the toa.ko file. After the compilation is successful, the file will be generated in the current directory.
    make
    # Load the TOA module.
    insmod toa.ko
    # Copy the TOA module to the kernel module directory.
    cp toa.ko /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko
    # Configure the TOA module to load automatically at system startup
    echo "insmod /lib/modules/`uname -r`/kernel/net/netfilter/ipvs/toa.ko" >> /etc/rc.local
    3. Run the following command to check whether the loading is successful:
    lsmod | grep toa
    If you see "TOA" in the message, the module is loaded successfully:
    

    Step 3: Verify the configuration

    You can verify the configuration by building a TCP server to receive client requests from another server. See the sample:
    1. On the current server, create an HTTP server in Python to act as a TCP server:
    # Use python2
    python2 -m SimpleHTTPServer 10000
    
    # Use python3
    python3 -m http.server 10000
    2. Make another server work as a client to send requests:
    # Use curl to initiate an HTTP request, where the hostname and forwarding port of the L4 proxy is used.
    curl -i "http://a8b7f59fc8d7e6c9.example.com.edgeonedy1.com:10000/"
    3. If TOA is loaded, the real client address can be seen on the server:
    
    
    You can also get either the IPv4 or IPv6 address of the client by following the steps above.
    For origin IPv4 addresses, get the client IPv4 address.
    For origin IPv6 addresses, get the client IPv6 address.
    If you need to get both IPv4 and IPv6 addresses, modify the origin's business code while loading the TOA module as instructed here.

    Getting Both IPv4 and IPv6 Addresses

    Note:
    This section provide guidance on how to get both IPv4 and IPv6 addresses by modifying the business code of the origin.
    The origin can listen on requests in either of the following methods:
    1. Use the structure struct sockaddr_in to listen on IPv4 addresses.
    2. Use the structure struct sockaddr_in6 to listen on IPv6 addresses.

    Sample code

    Listen on IPv4 addresses
    Listen on IPv6 addresses
    
    C
    Java
    #include <sys/socket.h>
    #include <stdio.h>
    #include <unistd.h>
    #include <netinet/in.h>
    #include <memory.h>
    #include <arpa/inet.h>
    
    int main(int argc, char** argv) {
    int l_sockfd;
    // The server address is an IPv4 address.
    struct sockaddr_in serveraddr;
    // In this case, the client address must adopt the IPv6 structure.
    struct sockaddr_in6 clientAddr;
    int server_port = 10000;
    
    memset(&serveraddr, 0, sizeof(serveraddr));
    // Create a socket.
    l_sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (l_sockfd == -1){
    printf("Failed to create socket.\\n");
    return -1;
    }
    // Initialize the server.
    memset(&serveraddr, 0, sizeof(struct sockaddr_in));
    serveraddr.sin_family = AF_INET;
    serveraddr.sin_port = htons(server_port);
    serveraddr.sin_addr.s_addr = htonl(INADDR_ANY);
    
    int isReuse = 1;
    setsockopt(l_sockfd, SOL_SOCKET,SO_REUSEADDR,(const char*)&isReuse,sizeof(isReuse));
    // Associate the socket and server address.
    int nRet = bind(l_sockfd,(struct sockaddr*)&serveraddr, sizeof(serveraddr));
    if(-1 == nRet)
    {
    printf("bind error\\n");
    return -1;
    }
    // Listen on the socket.
    listen(l_sockfd, 5);
    
    int clientAddrLen = sizeof(clientAddr);
    memset(&clientAddr, 0, sizeof(clientAddr));
    // Accept connections from the client.
    int linkFd = accept(l_sockfd, (struct sockaddr*)&clientAddr, &clientAddrLen);
    if(-1 == linkFd)
    {
    printf("accept error\\n");
    return -1;
    }
    // Modifications to make: Decide whether the client is an IPv4 or IPv6 address based on sin6_family.
    // AF_INET indicates that the client adopts IPv4. In this case, convert the client address pointer to struct sockaddr_in* and get the IPv4 address.
    // AF_INET6 indicates that the client adopts IPv6. In this case, use struct sockaddr_in6* to get the IPv6 address.
    if (clientAddr.sin6_family == AF_INET) {
    printf("AF_INET accept getpeername %s : %d successful\\n",
    inet_ntoa(((struct sockaddr_in*)&clientAddr)->sin_addr),
    ntohs(((struct sockaddr_in*)&clientAddr)->sin_port));
    }else if (clientAddr.sin6_family == AF_INET6){
    char addr_p[128] = {0};
    inet_ntop(AF_INET6, (void *)&((struct sockaddr_in6*)&clientAddr)->sin6_addr, addr_p, (socklen_t )sizeof(addr_p));
    printf("AF_INET6 accept getpeername %s : %d successful\\n",
    addr_p,
    ntohs(((struct sockaddr_in6*)&clientAddr)->sin6_port));
    }else{
    printf("unknow sin_family:%d \\n", clientAddr.sin6_family);
    }
    close(l_sockfd);
    return 0;
    }
    import java.io.IOException;
    import java.io.InputStream;
    import java.io.OutputStream;
    import java.net.InetAddress;
    import java.net.InetSocketAddress;
    import java.net.ServerSocket;
    import java.net.Socket;
    import java.net.SocketAddress;
    
    
    public class ServerDemo {
    
    
    /** If using the IPv4 address structure to build the service, use IPV4_HOST */
    public static final String IPV4_HOST = "0.0.0.0";
    
    
    /** If using the IPv6 address structure to build the service, use IPV6_HOST */
    public static final String IPV6_HOST = "::";
    
    
    public static void main(String[] args) {
    int serverPort = 10000;
    try (ServerSocket serverSocket = new ServerSocket()) {
    // Setting address reuse
    serverSocket.setReuseAddress(true);
    // Bound server address and port, using IPv4 here
    serverSocket.bind(new InetSocketAddress(InetAddress.getByName(IPV4_HOST), serverPort));
    System.out.println("Server is listening on port " + serverPort);
    
    
    while (true) {
    // Accepting Client connections
    Socket clientSocket = serverSocket.accept();
    System.out.println("New client connected: " + clientSocket.getRemoteSocketAddress());
    
    
    // Processing Client requests
    handleClientRequest(clientSocket);
    }
    } catch (IOException e) {
    System.err.println("Failed to create server socket: " + e.getMessage());
    }
    }
    
    
    /**
    * Processing Function, site business implement, here is just an example
    * The purpose of this Function is to Return the Client's input verbatim to the Client
    */
    private static void handleClientRequest(Socket clientSocket) {
    try (InputStream inputStream = clientSocket.getInputStream();
    OutputStream outputStream = clientSocket.getOutputStream()) {
    
    
    // Reading the Data received from the Client
    byte[] buffer = new byte[1024];
    int bytesRead;
    while ((bytesRead = inputStream.read(buffer)) != -1) {
    // Reply the received Data to the Client as it is
    outputStream.write(buffer, 0, bytesRead);
    }
    
    
    } catch (IOException e) {
    // When the Client disconnects
    System.err.println("Failed to handle client request: " + e.getMessage());
    } finally {
    try {
    clientSocket.close();
    } catch (IOException e) {
    System.err.println("Failed to close client socket: " + e.getMessage());
    }
    }
    }
    }
    
    C
    Java
    #include <sys/socket.h>
    #include <stdio.h>
    #include <unistd.h>
    #include <netinet/in.h>
    #include <memory.h>
    #include <arpa/inet.h>
    
    int main(int argc, char **argv)
    {
    int l_sockfd;
    // The server address is an IPv6 address.
    struct sockaddr_in6 serveraddr;
    // The client address is an IPv6 address.
    struct sockaddr_in6 clientAddr;
    int server_port = 10000;
    
    memset(&serveraddr, 0, sizeof(serveraddr));
    // Create a socket.
    l_sockfd = socket(AF_INET6, SOCK_STREAM, 0);
    if (l_sockfd == -1){
    printf("Failed to create socket.\\n");
    return -1;
    }
    // Set the server address.
    memset(&serveraddr, 0, sizeof(struct sockaddr_in6));
    serveraddr.sin6_family = AF_INET6;
    serveraddr.sin6_port = htons(server_port);
    serveraddr.sin6_addr = in6addr_any;
    
    int isReuse = 1;
    setsockopt(l_sockfd, SOL_SOCKET,SO_REUSEADDR,(const char*)&isReuse,sizeof(isReuse));
    // Associate the socket and server address.
    int nRet = bind(l_sockfd,(struct sockaddr*)&serveraddr, sizeof(serveraddr));
    if(-1 == nRet)
    {
    printf("bind error\\n");
    return -1;
    }
    // Listen on the socket.
    listen(l_sockfd, 5);
    
    int clientAddrLen = sizeof(clientAddr);
    memset(&clientAddr, 0, sizeof(clientAddr));
    // Accept connection requests from the client.
    int linkFd = accept(l_sockfd, (struct sockaddr*)&clientAddr, &clientAddrLen);
    if(-1 == linkFd)
    {
    printf("accept error\\n");
    return -1;
    }
    // The client addresses received here are all stored in the IPv6 structure.
    // The IPv4 addresses are mapped to IPv6 addresses, for example, "::ffff:119.29.1.1".
    char addr_p[128] = {0};
    inet_ntop(AF_INET6, (void *)&clientAddr.sin6_addr, addr_p, (socklen_t )sizeof(addr_p));
    printf("accept %s : %d successful\\n", addr_p, ntohs(clientAddr.sin6_port));
    // Modifications to make: Use the macro definition IN6_IS_ADDR_V4MAPPED to decide whether the client IP is an IPv4-mapped IPv6 address.
    if(IN6_IS_ADDR_V4MAPPED(&clientAddr.sin6_addr)) {
    struct sockaddr_in real_v4_sin;
    memset (&real_v4_sin, 0, sizeof (struct sockaddr_in));
    real_v4_sin.sin_family = AF_INET;
    real_v4_sin.sin_port = clientAddr.sin6_port;
    // The last four bytes represent the IPv4 address of the client.
    memcpy (&real_v4_sin.sin_addr, ((char *)&clientAddr.sin6_addr) + 12, 4);
    printf("connect %s successful\\n", inet_ntoa(real_v4_sin.sin_addr));
    }
    close(l_sockfd);
    return 0;
    }
    import java.io.IOException;
    import java.io.InputStream;
    import java.io.OutputStream;
    import java.net.InetAddress;
    import java.net.InetSocketAddress;
    import java.net.ServerSocket;
    import java.net.Socket;
    import java.net.SocketAddress;
    
    public class ServerDemo {
    
    /** If using the IPv4 address structure to build the service, use IPV4_HOST */
    public static final String IPV4_HOST = "0.0.0.0";
    
    /** If using the IPv6 address structure to build the service, use IPV6_HOST */
    public static final String IPV6_HOST = "::";
    
    public static void main(String[] args) {
    int serverPort = 10000;
    try (ServerSocket serverSocket = new ServerSocket()) {
    // Setting address reuse
    serverSocket.setReuseAddress(true);
    // Bound server address and port, using IPv4 here
    serverSocket.bind(new InetSocketAddress(InetAddress.getByName(IPV6_HOST), serverPort));
    System.out.println("Server is listening on port " + serverPort);
    
    while (true) {
    // Accepting Client connections
    Socket clientSocket = serverSocket.accept();
    System.out.println("New client connected: " + clientSocket.getRemoteSocketAddress());
    
    // Processing Client requests
    handleClientRequest(clientSocket);
    }
    } catch (IOException e) {
    System.err.println("Failed to create server socket: " + e.getMessage());
    }
    }
    
    /**
    * Processing Function, site business implement, here is just an example
    * The purpose of this Function is to Return the Client's input verbatim to the Client
    */
    private static void handleClientRequest(Socket clientSocket) {
    try (InputStream inputStream = clientSocket.getInputStream();
    OutputStream outputStream = clientSocket.getOutputStream()) {
    
    // Reading the Data received from the Client
    byte[] buffer = new byte[1024];
    int bytesRead;
    while ((bytesRead = inputStream.read(buffer)) != -1) {
    // Reply the received Data to the Client as it is
    outputStream.write(buffer, 0, bytesRead);
    }
    
    } catch (IOException e) {
    // When the Client disconnects
    System.err.println("Failed to handle client request: " + e.getMessage());
    } finally {
    try {
    clientSocket.close();
    } catch (IOException e) {
    System.err.println("Failed to close client socket: " + e.getMessage());
    }
    }
    }
    }

    console output result

    Server is listening on port 10000
    New client connected: /127.0.0.1:50680
    New client connected: /0:0:0:0:0:0:0:1:51124
    New client connected: /127.0.0.1:51136

    References

    Monitoring TOA Running Status

    To ensure execution stability, this kernel module allows you to monitor status. After inserting the toa.ko kernel module, you can monitor the TOA working status in either of the following ways.
    cat /proc/net/toa_stats
    This figure shows the TOA running status:
    
    The monitoring metrics are described as follows:
    Metric
    Description
    syn_recv_sock_toa
    Receives connections with TOA information.
    syn_recv_sock_no_toa
    Receives connections without TOA information.
    getname_toa_ok
    This count increases when you call getsockopt and obtain the source IP address successfully or when you call accept to receive client requests.
    getname_toa_mismatch
    This count increases when you call getsockopt and obtain a source IP address that does not match the required type. For example, if a client connection contains a source IPv4 address whereas you obtain an IPv6 address, the count will increase.
    getname_toa_empty
    This count increases when the getsockopt function is called in a client file descriptor that does not contain TOA.
    ip6_address_alloc audio/video proxy
    Allocates space to store the information when TOA obtains the source IP address and source port saved in the TCP data packet.
    ip6_address_free audio/video proxy
    When the connection is released, TOA will release the memory previously used to save the source IP and source port. If all connections are closed, the total count of ip6_address_alloc for each CPU should be equal to the count of this metric.
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support