Daniel Hoelbling-Inzko talks about programming

Amazon S3 uploads fail in numerous ways for no apparent reason

Today I spent almost 30 minutes trying to debug a problem where one of our Servers was not correctly synchronizing files to Amazon S3.

First thing I tried was manually doing a s3cmd put of the files in question, and was immediately greeted by numerous connection reset by peer error messages. s3cmd is so smart, it even automatically retries for 5 times while throttling the upload bandwith - still to no avail. When down to 5 kb/s upload speed I started getting broken pipe error messages.

Well, obviously our server has more than 5 kb/s uplink so I suspected someone broke something on the network level, but everything else was working fine.

The final clue then came when I was running `s3cmd ls` on my bucket where I finally got a meaningful error message: the difference between the request time and the current time is too large. Huh? Yes! Doing a date on the server revealed that the server was almost 20 minutes behind the current time, thus the SSL connection to S3 could not be established.

Turns out we had not enabled ntp and so the server clock kept drifting for a couple of months. The solution was straightforward:

sudo ntpdate ntp.ubuntu.com
sudo apt-get install ntp

The first does an immediate update (only installing ntp will take some time to sync the clock back up, so doing a manual update fixes your problem immediately), and then it installs ntp to prevent this from happening in the future.

Filed under tools

Installing Eclipse on Mac OSX

Most Mac applications come bundled as a .app package that you simply have to drag&drop into your Applications for them to show up in Launchpad and Alfred. Or, if an application is a bit more complex you usually get a .dmg file that then installs like any normal MSI package you are used to from windows.

Not Eclipse for OSX. The Eclipse project maintainers do provide a OSX binary, but it doesn't look like most OSX .app packages in that the extracted binaries are a folder that then contains a .app.

If you only copy over the .app file you'll not be able to start Eclipse as it's just a starter script and the real Eclipse is living in the extracted Zip. I used to just start eclipse wherever I unpacked it, but apparently the right way to install it is to simply copy the whole extracted eclipse folder over to your Applications. Even though the .app is not in /Applications directly OSX picks it up and it shows up in Launchpad.

Filed under programmierung, java

When to use raw() and when to use .html_safe

What rails does very nicely is protect you from XSS attacks through aggressively HTML encoding anything you write between <%= %>. But there is one caveat: At times you may really want to render HTML from a string. So you need to tell rails not to escape your HTML in that case.

There are two methods of telling rails that a string is safe and should not be escaped: raw and .htmlsafe

And both do the same. They mark the string as safe (through the use of the html safe buffer) and rails will not encode it any more. The main difference between the two: nil.

If you are doing things like: "<img src='#{..}' />".html_safe .html_safe is totally fine as the string will never be nil, but if you are dealing with strings that may be nil .html_safe will break since there is no .html_safe method on the nil object. (For example if you are loading something from a config value or the database) In that case using raw(...) will just ignore the string instead of raising an exception.

As always with these things: raw and html_safe make it very easy to introduce XSS attack vectors into your application. So use them wisely and only on strings you are sure to be safe.

Filed under programmierung, rails, ruby

How to expire all active Sessions in Rails 3

I know this sounds like a very simple task to do, but since I just spent half an hour reading up on how Sessions in Rails work I decided it's time to put this up so I can Google it again next time :)

First off, you need to know how you are storing sessions in your application. Rails supports 3 types of session storage: Stored on Disk, in the Database or through Cookies in the client. You can check which one you are using in /config/initializers/session_store.rb.

Cookie based session storage

Rails 3 defaults to storing the session in the client using a session cookie. This means that the user_id along with all the data you put into the session hash is serialized into the cookie and sent to the client. It's also not encrypted, only BASE64 encoded so if you are storing anything sensitive in there you are doing it wrong.

But for simple things like the current user_id the cookie based session store is just fine and also a lot faster than the alternatives.

Expiring the cookie though is a bit more involved since you can't reach out to all clients and delete their cookies at once. But, and that's the important part for what I was doing: This cookie is signed with a SHA-512 digest using a secret key that is only present on the server. So the cookie cannot be tampered with on the client, and this is also your avenue of attack when trying to expire all cookies:

Simply change the secret that is used to sign the cookies. All previous cookies are invalidated as their digest is no longer valid.

Doing so is simple, first generate a new secret using rake:

$ rake secret 10dfec4781b682762a731a5e88af78521fc3e0f...

Then copy/paste that new secret into your config/initializers/secret_token.rb:

MyApp::Application.config.secret_token = '10dfec4781b682762a731a5e88...'

Once done deploy the application and all existing sessions are invalid at once.

Database backed session storage

If you are using the Database to store the session it's rather trivial to expire all existing sessions using rake:

rake db:sessions:clear

File based session storage

Like with the database simply run the following rake command:

rake tmp:sessions:clear

Hope this helps..

Filed under programmierung, rails, ruby

Optional locals in Rails partial templates

In Rails it is very advisable to not use any instance variables inside your partials if you want to re-use that partial in a different context.

It is much better to simply leverage locals that you pass into your render like this:

That way you can simply use posts inside the template without having to rely on @posts coming from the controller (and it's also not dependant on controller code any more but rather only on data inside the view that's calling it.. very handy).

But, as always there are times when you have additional locals that are optional to only some contexts. In my case I had Kaminari paginate the list I was passing to the partial, but in some other views the list was not be paginated. Turns out you can't do the usual rails style if foo here because the variable is simply not defined and Ruby will throw an error (unlike with @variables in Rails).

That's where the defined? method comes in very handy (as suggested by Thorsten Ball):

If you simply want to assign a default value to a local you can use the ||= operator:

This enables me to use the partial in several ways without having to re-introduce the locals at every corner:

Filed under programmierung, rails, ruby

Explicit path helpers with STI in Rails

I am currently working on an application that makes use of Single-Table-Inheritance (STI) for some of it's models.

STI in Rails is quite simple, you simply subclass a model and add the type column to your table like this:

And the accompanying migration:

Now that that's out of the way, Rails url helpers will automagically use the right route depending on the class you feed to it's helpers. Here's an example:

You get the idea, rails helpers are smart about this. Only problem: How about the cases where you need an explicit helper like edit_person_path?

This is where it get's tricky:

Using the explicit helper rails will not do any STI magic and simply return the path you asked for. What I really needed was for a helper that points to /persons/:id/edit in case it's a Person record and to /students/:id/edit for students. After some digging (Rails STI isn't really the hottest topic on Google it seems) I found the polymorphic_path method that does exactly this:

Filed under rails, rails-2, ruby, ruby-2

Removing delete and destroy in Rails models

Today I had an interesting feature request to implement: Get rid of all deletions and replace them with a deleted flag in the DB.

It's the usual story: Nobody really does deletes but rather everything is put into the DB in case it's needed at some later point. Unfortunately I didn't know about this particular feature until after I had finished most of the coding for the Rails application so going through the code and removing all destroy code from controllers was pretty much out of the question.

Instead I remembered reading about Modules in Ruby and how they can bolt on functionality to Classes.

Turns out it's totally trivial to remove deletion from Rails Models with 10 odd lines of code in a completely transparent an unobtrusive way:

module NotDeleteable
  def destroy
    unless self.respond_to? :deleted
      raise MissingMigrationException

self.update_attribute :deleted, true end

def delete self.destroy end

def self.included(base) base.class_eval do default_scope where( :deleted => false ) end end end

class MissingMigrationException < Exception def message "Model is lacking the deleted boolean field for NotDeleteable to work" end end

This will override the default destroy/delete method provided by ActiveRecord::Base and also install a default_scope into the Model class so Rails will by default append WHERE deleted = false to all SQL queries made through the ActiveRecord Query Interface

You use this by simply including this module inside your Model class:

class User < ActiveRecord::Base
  include NotDeleteable
Did I mention that I really like Ruby?

Word of warning: I have no clue if I am breaking :dependant => :destroy on ActiveRecord relations in any way, but I suspect it should still be alright.

Update: The original code had an issue where you could not mark a record that is invalid as deleted. This was due to the fact that I was using save instead of update_attribute.

Filed under programmierung, rails, ruby

A word of advice on Bundler and Gems from git

At times (especially after Rails version upgrades) some Gems don't work out of the box so I usually spend some time on GitHub looking through the network graph for a fork that fixes the issue.

This works great and you usually end up with a well working plugin to continue working. Bundler supports this by loading the gem from the git source:

gem 'rails', :git => 'https://github.com/rails/rails.git'

Only problem with this approach: Forks are usually a moving target in regards to their longevity. People tend to delete them once the pull request gets merged, or simply forget about them.

Neither is desirable and today it bit me for the first time, I had referenced a fork that got deleted but contained a important fix for my project on a gem that was not yet 3.1 ready. Fortunately the project has moved along and the master already included the fix (and didn't introduce new issues). But it sure taught that I should always keep my own fork of code I decide to depend upon.

Filed under uncategorized

Make GNU screen xterm-256color work on OSX

I just ran into this and spend like 2 hours with Linux genius Jam trying to figure out why in the heck I could run Vim in 256 colors mode on my server while once I started screen it didn't work any more.

The issue was two fold. a) My local Terminal.app was reporting itself as xterm-color instead of xterm-256color. You have to update this setting in your Terminal app here:


Once done you only need to edit your .screenrc to include the following 3 lines:


# terminfo and termcap for nice 256 color terminal
# allow bold colors - necessary for some reason
attrcolor b ".I"
# tell screen how to set colors. AB = background, AF=foreground
termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'
# erase background with current bg color
defbce "on" 
# set TERM
term screen-256color-bce

Problem is: Even if you set your .screenrc correctly, it won't matter if your terminal is not reporting the correct version string in the first place..

Filed under tools

New team member on dotless: Luke Page

You may have noticed that my blog is filling itself slowly with stuff about Ruby and Unix in general. This has to do with the fact that for now 3 months I am working on Rails full time with very little to no .NET work in between.

While this is awesome for me and I am really enjoying it - it also means that I don't have a .NET dev environment available at work and quickly merging in a pull request and testing it has become quite a hassle. So dotless has had quite some open tickets that where already fixed but where not yet merged into the mainline due to me lagging behind on responding to pull requests.

Fortunately, most/all pull requests have come from our very active Luke Page who has been busily fixing bugs and contributing features - so adding him to the dotless core team is a logical choice to cut down on the lag I or James where inducing into the process.

Since James felt the same we are happy to announce that Luke Page is now part of the dotless team with commit access and everything. So, welcome onboard Luke! Thanks for being part of dotless :)

My Photography business


dynamic css for .NET