Skip to content

Obviate.io

To anticipate and prevent

  • Home
  • About Us
  • History
  • Privacy Policy
  • Toggle search form

Identical Docker, Identical OS – Crashing on 50% of the Hardware

Posted on 2016-02-16 By Jon No Comments on Identical Docker, Identical OS – Crashing on 50% of the Hardware

sad-mac-icon
sad-mac-icon

Gather round l’il ones and let old man Jon tell you a little story about the time he learned to never say never. Twas not terribly long ago when he was trying to spin up a fleet of Squid Proxies. Being the hip DevOps guru he is, the proxies were stored safely inside of Docker containers. Those containers were all pulled from the same single source, so they were identical (as one would expect from Docker). It also happens that these Docker containers were being spun up on a pair of brand new AWS EC2 instances. Same AMI, same bootstrap script, same software updates. Everything on every container should be identical. However when it came time to spin up the containers… 50% of them failed. W. T. F?

The error was from Squid which crashed on startup. Of course being Docker, when Squid dies, the container dies. That’s expected functionality. But why is Squid dying?

Illegal instruction (core dumped)

That might seem like a vague error message, but for those who’ve been around a while it does point us in the right direction. Messages like “illegal instruction” typically indicate that the application is trying to tell the CPU to do something that it can’t do. In the modern era of JavaScript, Ruby, and Python, that’s almost never seen. However Squid is a bit more hardcore and is written in C/C++ where you can actually tell the CPU to do something “illegal”.

Screen Shot 2016-02-15 at 9.12.47 PM
Screen Shot 2016-02-15 at 9.12.47 PM

It was confusing as all hell, that’s for sure. I’ve never seen a case where identical application stacks failed on only one of two brand new instances. However astute readers will have deduced that “identical” is actually “identical software”, but not necessarily identical hardware. In fact after only a little googling I found a Fedora bug report which was spot on for my troubles. The short version is that my particular version of Squid has a very particular bug in it. It also just so happens that AWS launched one instance on Intel Xeon Processor E5 v2 hardware (which was the crashing machine) and one instance on Intel® Xeon® Processor E5 v3 hardware (which worked).

AWS, Hardware Tags:core dump, docker, hardware not software, illegal instructionc, indentical, segfault, software not hardware, squid proxy, Ubuntu, v2 vs v3

Post navigation

Previous Post: Winter Urban Garden: Salad time!
Next Post: My name is Josh, and San Francisco is complicated

More Related Articles

WordPress: Enabling SSH/SFTP Updates Linux
AWS SES + Postfix + DKIM Cloud
Installing & Configuring OpenDS 2.2 on Ubuntu 10.04 LDAP

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

February 2016
S M T W T F S
 123456
78910111213
14151617181920
21222324252627
2829  
« Jan   Mar »

Copyright © 2022 Obviate.io

Powered by PressBook Premium theme