Tigraine

Daniel Hoelbling-Inzko talks about programming

Amazon S3 uploads fail in numerous ways for no apparent reason

Today I spent almost 30 minutes trying to debug a problem where one of our Servers was not correctly synchronizing files to Amazon S3.

First thing I tried was manually doing a s3cmd put of the files in question, and was immediately greeted by numerous connection reset by peer error messages. s3cmd is so smart, it even automatically retries for 5 times while throttling the upload bandwith - still to no avail. When down to 5 kb/s upload speed I started getting broken pipe error messages.

Well, obviously our server has more than 5 kb/s uplink so I suspected someone broke something on the network level, but everything else was working fine.

The final clue then came when I was running `s3cmd ls` on my bucket where I finally got a meaningful error message: the difference between the request time and the current time is too large. Huh? Yes! Doing a date on the server revealed that the server was almost 20 minutes behind the current time, thus the SSL connection to S3 could not be established.

Turns out we had not enabled ntp and so the server clock kept drifting for a couple of months. The solution was straightforward:

sudo ntpdate ntp.ubuntu.com
sudo apt-get install ntp

The first does an immediate update (only installing ntp will take some time to sync the clock back up, so doing a manual update fixes your problem immediately), and then it installs ntp to prevent this from happening in the future.

Filed under tools

My Photography business

Projects

dynamic css for .NET

Archives

more