Prevent Docker containers from crashing with error 137 (2024)

Docker systems can be used fora wide range of applications, from setting up development environments to hosting web instances.

Live Docker containers that crash often can end up ruiningtheirpurpose. As a result, a majorconcern faced by Docker providers is ensuring the containeruptime.

Containers crash due to many reasons, the main issue being lack of enoughmemory. During a crash, container will show an exit code that explains the reason for its crash.

Today we’ll see what causes Docker ‘Exited (137)’ crash message and how to fix it.

What causes error 137 in Docker

Docker crashes are often denoted by the message ‘Exited’ in the container ‘STATUS’ when listing the container using ‘docker ps -a’ command.

Error137 in Docker denotes that the container was ‘KILL’ed by ‘oom-killer’ (Out of Memory). This happens when there isn’t enough memory in the container for running the process.

‘OOM killer’ is a proactive process that jumps in to save the system when itsmemory level goes too low, by killing theresource-abusive processes to free up memory for the system.

Here is a snippet that shows that the Mysql container Exited with error 137:

When the MySQL process running in the container exceeded its memory usage, OOM-killer killed the container and it exited with code 137.

[ Running a Docker infrastructure doesn’t have to be hard, or costly. Get world class Docker management services at affordable pricing.]

How to debugerror 137 in Docker

Each Docker container has a log file associated with it. These log files store all the relevant information and updates related to that container.

Examining the container log file is vital in troubleshooting its crash. Details of the crash can be identified by checking the container logs using ‘docker logs’ command.

For the MySQL container that crashed, the log files showed the following information:

The logs in this case clearly shows that the MySQL container was killeddue to the mysqld process taking up too much memory.

Reasonsfor ‘Out of memory’ error 137 in Docker

In Docker architecture, the containers are hosted in a single physical machine. Error 137 in Docker usually happens due to 2 main out-of-memory reasons:

1. Docker container has run out of memory

By default, Docker containers use the available memory in the host machine. To prevent asingle container from abusing the host resources, we set memory limits per container.

But if the memory usage by the processes in the container exceeds this memory limit set for the container, the OOM-killer would kill the applicationand the container crashes.

An application can use up too much memory due to improper configuration, service not optimized, high traffic or usage or resource abuse by users.

2. Docker hosthasno free memory

The memory limit that can be allotted to the Docker containers is limited by the total available memory in the host machine which hosts them.

Many often, when usage and traffic increases, the available free memory may be insufficientfor all the containers.As a result, containers may crash.

[ Never let your business be affected by crashing containers!Our Docker experts take care of your infrastructure and promptly resolves all container issues.]

How to resolveerror 137 in Docker

When a Docker container exits with OOM error, it shows that there is a lack of memory. But, the first resort should not be to increase the RAM in the host machine.

Improperly configuredservices, abusive processes or peak traffic can lead to memory shortage. So the first option is to identify the cause of this memory usage.

After identifying the cause, the following corrective actions can be done in the Docker system to avoid further such OOM crashes.

1. Optimize the services

Unoptimized applicationscould take up more than optimalmemory.For instance, an improperly configured MySQL service can quickly consume the entire host memory.

So, the first step is to monitor the application running in the container and to optimize the service. This can be done by editing the configuration file or recompiling the service.

2. Mount config files from outside

It is always advisable to mount theconfig files of services from outside the Docker container. This will allow to edit them easily without recompiling the Docker image.

For instance, in MySQL docker image, “/etc/mysql/conf.d” can be mounted as a volume. Any configuration changes for MySQL service can be done without affecting thatimage.

3. Monitorthe container usage

Monitoring the container’s memory usage to detectabusive users, resource-depleted processes, traffic spikes, etc. is very vital in Docker system management.

Depending on the traffic and resource usage of processes, memory limitsfor the containers can be changed to suit their business purpose better.

4. Add more RAM to the host machine

After optimizing the services and setting memory limits, if thecontainers are running at their maximum memory limits, then we should add more RAM.

Adding more RAM and ensuring enough swapmemory in the host machine would help the containers to utilize that memory whenever there is a memory crunch.

In short..

Today we saw how to fix error 137 in Dockerusinga systematic debugging method.But there may be scenarios where the error code may not be displayed, especially when the container is launched from some shell script.

At Bobcares, examining Docker logs, optimizing applications, limiting resource usage for containers, monitoring the traffic, etc. areroutinely done to avoid crashes.

Before deploying Docker containers for live hosting or development environment setup, we perform stress test by simulating the estimated peak traffic.

This helps us to minimize crashes in the live server.If you’d like to know how to manage your Dockerresources efficiently for your business purpose, we’d behappy to talk to you.