tencent cloud

Feedback

Use Exit Code to Identify Pod Exceptions

Last updated: 2025-01-23 17:59:44
    This document describes how to use exit codes to troubleshoot pod issues.

    Querying Pod Exceptions

    Run the following command to query pod exceptions:
    kubectl describe pod <pod name>
    The returned result is as follows:
    Containers:
    kubedns:
    Container ID: docker://5fb8adf9ee62afc6d3f6f3d9590041818750b392dff015d7091eaaf99cf1c945
    Image: ccr.ccs.tencentyun.com/library/kubedns-amd64:1.14.4
    Image ID: docker-pullable://ccr.ccs.tencentyun.com/library/kubedns-amd64@sha256:40790881bbe9ef4ae4ff7fe8b892498eecb7fe6dcc22661402f271e03f7de344
    Ports: 10053/UDP, 10053/TCP, 10055/TCP
    Host Ports: 0/UDP, 0/TCP, 0/TCP
    Args:
    --domain=cluster.local.
    --dns-port=10053
    --config-dir=/kube-dns-config
    --v=2
    State: Running
    Started: Tue, 27 Aug 2019 10:58:49 +0800
    Last State: Terminated
    Reason: Error
    Exit Code: 255
    Started: Tue, 27 Aug 2019 10:40:42 +0800
    Finished: Tue, 27 Aug 2019 10:58:27 +0800
    Ready: True
    Restart Count: 1
    Exit Code is the status code of the last container exit. If it is not 0, the container exited due to an exception. You can use the exit code to further troubleshoot the problem.

    Exit Codes

    A valid exit code is between 0 and 255.
    0 means the container exited normally.
    If the container exited due to an external signal, the exit code is between 129 and 255. For example, if the operating system sent kill -9 or ctrl+c as the termination signal, the status is SIGKILL or SIGINT.
    If the container exited due to an internal signal, the exit code is between 1 and 128. However, in some circumstances, the exit code might be between 129 and 255.
    If the specified exit code has a value outside the 0-255 range, such as exit(-1), it is automatically translated to a value in the 0-255 range. If the exit code is specified as code, it is translated as follows:
    If the exit code is negative:
    256 - (|code| % 256)
    If the exit code is positive:
    code % 256

    Typical Exit Codes

    137: indicates that the process was killed by SIGKILL. Possible reasons are:
    Pod memory reached resources.limits, such as Out of Memory (OOM). Pod resource limits are implemented by using Linux cgroup. If the memory of a pod reaches its limit, cgroup forces it to stop (with a similar effect to kill -9). If you use describe pod, you can see the value of Reason is OOMKilled.
    If the host does not have sufficient resources (OOM), the kernel stops some processes to free up the memory.
    Note:
    If the process is stopped due to OOM, cgroup, or the host, you can find relevant records in system logs:
    Ubuntu system logs are stored in /var/log/syslog, whereas CentOS system logs are stored in /var/log/messages. You can run the journalctl -k command to view system logs in both operating systems.
    livenessProbe failed, which causes kubelet to stop the pod.
    Pod stopped by a trojan process.
    1 and 255: indicates common issues. Check container logs for further troubleshooting. For example, this could be the result of exit(1) or exit(-1). -1 is translated to 255.

    Standard Linux Interruption Signals

    Linux programs send an exit code when they are interrupted by external signals. The value of the exit code is the value of the interrupt signal plus 128. For example, the value of SIGKILL is 9, so the program exit code is 9 + 128 = 137. For more standard interrupt signals, see the following table:
    Signal
    Status Code Value
    Action
    Description
    SIGHUP
    1
    Term
    Hangup detected on controlling terminal or death of controlling process
    SIGINT
    2
    Term
    Interrupt from keyboard
    SIGQUIT
    3
    Core
    Quit from keyboard
    SIGILL
    4
    Core
    Illegal Instruction
    SIGABRT
    6
    Core
    Abort signal from abort(3)
    SIGFPE
    8
    Core
    Floating-point exception
    SIGKILL
    9
    Term
    Kill signal
    SIGSEGV
    11
    Core
    Invalid memory reference
    SIGPIPE
    13
    Term
    Broken pipe: write to pipe with no readers; see pipe(7)
    SIGALRM
    14
    Term
    Timer signal from alarm(2)
    SIGTERM
    15
    Term
    Termination signal
    SIGUSR1
    30,10,16
    Term
    User-defined signal 1
    SIGUSR2
    31,12,17
    Term
    User-defined signal 2
    SIGCHLD
    20,17,18
    Ign
    Child stopped or terminated
    SIGCONT
    19,18,25
    Cont
    Continue if stopped
    SIGSTOP
    17,19,23
    Stop
    Stop process
    SIGTSTP
    18,20,24
    Stop
    Stop typed at terminal
    SIGTTIN
    21,21,26
    Stop
    Terminal input for background process
    SIGTTOU
    22,22,27
    Stop
    Terminal output for background process
    

    C/C++ Exit Codes

    /usr/include/sysexits.h provides standardized exit codes for C and C++. These codes are described in the following table:
    Definition
    Status Code
    Description
    #define EX_OK
    0
    successful termination
    #define EX__BASE
    64
    base value for error messages
    #define EX_USAGE
    64
    command line usage error
    #define EX_DATAERR
    65
    data format error
    #define EX_NOINPUT
    66
    cannot open input
    #define EX_NOUSER
    67
    addressee unknown
    #define EX_NOHOST
    68
    host name unknown
    #define EX_UNAVAILABLE
    69
    service unavailable
    #define EX_SOFTWARE
    70
    internal software error
    #define EX_OSERR
    71
    system error (e.g., can't fork)
    #define EX_OSFILE
    72
    critical OS file missing
    #define EX_CANTCREAT
    73
    can't create (user) output file
    #define EX_IOERR
    74
    input/output error
    #define EX_TEMPFAIL
    75
    temp failure; user is invited to retry
    #define EX_PROTOCOL
    76
    remote error in protocol
    #define EX_NOPERM
    77
    permission denied
    #define EX_CONFIG
    78
    configuration error
    #define EX__MAX 78
    78
    maximum listed value
    

    Status Code Reference

    For the description of more status codes, see the following table:
    Status Code
    Meaning
    Example
    Description
    1
    Catchall for general errors
    let "var1 = 1/0"
    Miscellaneous errors, such as "divide by zero" and other impermissible operations
    2
    Misuse of shell builtins (according to Bash documentation)
    empty_function() {}
    Missing keyword or command, or permission problem (and diff return code on a failed binary file comparison).
    126
    Command invoked cannot execute
    /dev/null
    Permission problem or command is not an executable
    127
    "command not found"
    illegal_command
    Possible problem with $PATH or a typo
    128
    Invalid argument to exit
    exit 3.14159
    exit takes only integer args in the range 0 - 255 (see first footnote)
    128+n
    Fatal error signal "n"
    kill -9 $PPID of script
    $? returns 137 (128 + 9)
    130
    Script terminated by Control-C
    Ctl-C
    Control-C is fatal error signal 2, (130 = 128 + 2, see above)
    255*
    Exit status out of range
    exit -1
    exit takes only integer args in the range 0 - 255
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support