tencent cloud

Feedback

COS Stress Test Guide

Last updated: 2022-06-28 14:53:10

    COSBench Overview

    COSBench is an open-source stress test tool developed by Intel for testing the performance of cloud object storage systems. As a cloud storage system compatible with S3 protocol, COSBench can be used to perform benchmark tests on the read/write performance of Tencent Cloud COS.

    System Environment

    We recommend you run COSBench on CentOS 7.0 or later. If you run it on Ubuntu, unexpected issues may occur.

    Performance Factors

    • CPU cores: A small number of CPU cores coupled with a large number of running workers is very likely to cause high overheads for context switching. Therefore, 32 or 64 cores are recommended for the test.
    • NIC: The outbound traffic from the server is limited by the NIC. To test the traffic for large files, an NIC above 10 GbE is recommended.
    • Request network linkage: Public network linkages vary in quality. Charges will incur for public network downstream traffic for downloads over the public network. Therefore, private network is recommended for access within the same region.
    • Test duration: In order to get a reliable value, a longer test period is recommended.
    • Test environment: The version of the JDK running on your program is also a key performance factor. For example, when testing on HTTPS, with an earlier JDK version, GCM bugs may occur for the encryption algorithm, as well as the locking issues for the random number generator.

    Directions

    1. Download COSBench 0.4.2.c4.zip from GitHub and decompress it on your server.
    2. Install the COSBench dependency library and run the following command.
    • For CentOS, run the following command to install the dependencies:

      sudo yum install nmap-ncat java curl java-1.8.0-openjdk-devel -y
      
    • For Ubuntu, run the following command to install the dependencies:

      sudo apt install nmap openjdk-8-jdk 
      
    1. Edit the s3-config-sample.xml file and configure a test job. The test job is divided into the following five stages.

      1. init: Creates a bucket.
      2. prepare: Uses worker threads to PUT (upload) objects in specified size for read in the main stage.
      3. main: Uses worker threads to read and write objects for a specified period of time.
      4. cleanup: Deletes the created objects.
      5. dispose: Deletes the bucket.

      The sample configuration is as shown below:

      <?xml version="1.0" encoding="UTF-8" ?>
      <workload name="s3-50M-sample" description="sample benchmark for s3">
      <storage type="s3" config="accesskey=AKIDHZRLB9Ibhdp7Y7gyQq6BOk1997xxxxxx;secretkey=YaWIuQmCSZ5ZMniUM6hiaLxHnxxxxxx;endpoint=http://cos.ap-beijing.myqcloud.com" />
      <workflow>
       <workstage name="init">
        <work type="init" workers="10" config="cprefix=examplebucket;csuffix=-1250000000;containers=r(1,10)" />
      </workstage>
       <workstage name="prepare">
        <work type="prepare" workers="100" config="cprefix=examplebucket;csuffix=-1250000000;containers=r(1,10);objects=r(1,1000);sizes=c(50)MB" />
      </workstage>
       <workstage name="main">
        <work name="main" workers="100" runtime="300">
          <operation type="read" ratio="50" config="cprefix=examplebucket;csuffix=-1250000000;containers=u(1,10);objects=u(1,1000)" />
          <operation type="write" ratio="50" config="cprefix=examplebucket;csuffix=-1250000000;containers=u(1,10);objects=u(1000,2000);sizes=c(50)MB" />
        </work>
      </workstage>
       <workstage name="cleanup">
        <work type="cleanup" workers="10" config="cprefix=examplebucket;csuffix=-1250000000;containers=r(1,10);objects=r(1,2000)" />
      </workstage>
       <workstage name="dispose">
        <work type="dispose" workers="10" config="cprefix=examplebucket;csuffix=-1250000000;containers=r(1,10)" />
      </workstage>
      </workflow>
      </workload>
      

    Parameter description

    ParameterDescription
    accesskey, secretkey Key information. Replace them with your own `SecretId` and `SecretKey`.
    cprefix Bucket name prefix, such as `examplebucket`.
    containers Value range for bucket names. A bucket name is made up of `cprefix` and `containers`, such as `examplebucket1` or `examplebucket2`.
    csuffix Your `APPID`. Note that `APPID` is prefixed with a -, such as `-1250000000`.
    runtime Duration of the stress test
    ratio Ratio of reads to writes
    workers Number of stress test threads
    4. Edit the `cosbench-start.sh` file and add the following parameter to the Java startup command line to disable S3 MD5 verification.
    -Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true
    
    1. Start the COSBench service.
    • For CentOS, run the following command:

      sudo bash start-all.sh
      
    • For Ubuntu, run the following command:

      sudo bash start-driver.sh &
      sudo bash start-controller.sh &
      
    1. Run the following command to submit the job.
      sudo bash cli.sh submit conf/s3-config-sample.xml
      

    Check the test status at http://ip:19088/controller/index.html (replace the IP in this link with the IP of your own testing server).

    You can see the five stages as shown below:

    7. The following example shows the performance tests of uploads and downloads in a Tencent Cloud CVM instance with 32 cores and 17 Gbps private network bandwidth in Beijing region. The test includes the following two stages.
    1. prepare: 100 workers threads are run to upload 1,000 objects (50 MB each).
    2. main: 100 workers are run to read and write objects in parallel for 300 seconds.

    The test results after the above two stages are as shown below.

    8. Run the following command to stop the test.

    sudo bash stop-all.sh
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support