Tigraine

Daniel Hoelbling-Inzko talks about programming

Mongoid .length on query ignores limit

I just ran into this and wanted to share this quite puzzling behavior:

20.times { Fabricate(:post) }
Post.scoped.limit(10).length.should eq(10)
# => Test failed
#    expected: 12
#         got: 20

Weird right? But once you look at MongoDB's capabilities and the Mongoid implementation you'll quickly discover that this is only kind of a semi-bug :). MongoDB supports an .count() command that takes a filter and returns the number of matches documents. This command is being issued by Mongoid whenever you call .length on a criterion (since it's much cheaper than executing the actual query and counting the returned objects).

So don't write tests like this:

# method under test:
def self.most_played
  self.order_by(:play_count.desc).limit(12)  
end

# test
it 'returns the 12 most played items' do
  Video.most_played.length.should eq(12)
end

This test will fail, even though the method is doing everything perfectly right. I'd even say that in a regular ActiveRecord setting this would be a passing test. But since I am utilizing the delayed execution functionality of Mongoid most_played method is only returning a Criteria object that represents the query and can be composed into a more complicated query.

This causes .length to be executed on Criteria where it causes Mongoid to call the .count() method instead of running .length on Array. Since limit is no query parameter it will not be passed to MongoDB and the returning count will correctly be equal to the total number of Documents in the collection.

The more correct way to write that test is to actually force Mongoid to execute it's query and count the result:

it 'returns the 12 most played items' do
  Video.most_played.to_a.length.should eq(12)
end
Filed under mongoid, rails, ruby

German programming-friendly keyboard layout for Mac OSX

This may sound totally hilarious to people used to the english layout, but german programmers use shortcuts like Shift + 8 to write a [ and ALT-GR + 8 for a {. (Oh and semicolons are only available through shift - yay for C-style languages!) The main reason for this is that we have these useless little umlauts üöä that sit right where the useful keys for a programmer would be, were it not for the german keyboard.

But after years of arm-twisting and cursing you get used to that and develop a reasonable degree of efficiency even with a absolutely unsuitable keyboard layout. Only problem: these shortcuts are radically different on the Mac! (And breaking the built-in keyboard of your Laptop is not really an option as others suggest)

Instead of Shift 8 for a brace it's ALT + 5, and the backslash is skillfully hidden behind ALT+Shift+6 yuck!

So I tried for a few days and then I decided to stop the madness! Instead of learning to use yet another crappy german keyboard layout I started to use the standard english one. And it's awesome, once you get used to it feels absolutely natural, and it also shows that all programming languages have been written to be easy on people with english keyboard layouts.

Only problem: I was severly handicapped when writing german emails to customers or writing german translation texts. I mean, my friends and colleagues can read written out umlauts (ä = au ..) just fine, but it looks totally stupid when writing something official and well phrased and you are misspelling stuff (ß can be sloppily substituted by ss, but there are cases where ss is orthographically wrong and in other cases ß would be wrong and ss right).

So I investigated and found this wonderful Mac program called Ukulele that allows you to write your own keyboard layout files for OSX. I took the standard US layout and altered it to incorporate the german umlaut keys where they are actually printed on the keys, but only if you hit ALT. So ALT + ; turns into a ö etc..

Here a quick overview of the english layout (notice how conveniently located all those symbols are):

alt

And once you hold down ALT:

alt

Since most people I mention this to usually give me a: "wow that's a really good idea" I decided to simply upload the .keylayout file so people don't have to bother with Ukulele themselves.

You can download the keylayout here: US-daniel.keylayout.

Installation:

Download the file and place it in ~/Library/Keyboard Layouts/, logout and log back into the machine. Now open your System preferences (Sprache & Text) and search for U.S. Tick the checkbox next to it and you can now select the U.S. Keyboard from the OSX menu bar.

Tipp: If you have the screen real estate to share and are still learning the english keyboard layout it's really useful to enable the "Keyboard overview" (Tastaturübersicht anzeigen in german). You can see in real-time what keys are mapped to what buttons on your keyboard while holding down modifier keys.

Filed under osx, programming

Failing silently: Carrierwave ignores store_dir

Carrierwave, my favorite file upload framework of choice in Rails requires something along the lines of 5 lines of code to set up a file upload. The uploader can then be mounted on a model and receive file_field uploads from forms. Especially with the numerous storage backends and the not so common support for ORM frameworks besides ActiveRecord Carrierwave is right on par with other important Gems like Devise.

But then, after having used it upwards of 20 times, just when you think you remember the syntax without having to check the site something unexpected puzzles you:

In my case that was a file upload that was technically working, but seemed to completely ignore the options specified in the uploader. The store_dir was ignored, the different versions were simply never called - but the file upload worked like a charm.

I thought I did everything right (the uploader code looked just fine) and apparently I was uploading files just fine - but it didn't work.

After some investigation and re-doing everything following the getting-started instructions the following code in my model caught my eye:

  mount_uploader :menu_image

Since my uploader was called MenuImageUploader I thought: well that looks reasonable.

Turns out I was wrong (shocking I know) .. Carrierwave requires you to explicitly specify the desired uploader class instead of deducing it from the name.

  mount_uploader :menu_image, MenuImageUploader

Suddently everything started working again, yay. So: If you ever need to just simply upload a file that has no special storage directory or other processing going on - just mount a uploader without a uploader class. Carrierwave will default to Carrierwave::Uploader::Base for you. And if that wasn't you intention - it will still screw you over :).

Maybe I should raise this as an issue with the developers. At least a simple logger.warn would be in order in cases like these.

Filed under rails, ruby, carrierwave

Mongoid i18n and fallback locales

I may have hinted at it once or twice that I really love how Rails makes Internationalization (I18n) really simple (in stark contrast to what I had to do back then in .NET). But one thing Rails I18n will not do for you is translating the models in your database, which is to me one of the major problems I often face at work when doing I18n. (Of course there are gems that alleviate that pain)

So today I was sitting down to write a system design for a new application at work that has to be localized in a lot of different languages while still being fast. After thinking a bit about the problem I decided that doing so in ActiveRecord would be cumbersome to say the least, and I decided for a Document-Database approach. Since I know MongoDB best I went out and looked at the Mongoid documentation on how to best integrate transparent localization into the system (fully expecting that I'd have to implement this myself).

Well, Mongoid blew my mind:

class Post
  include Mongoid::Document

  field :title, localized: true
end

That's it, Mongoid will now transparently store the title field in a hash and return the correct value depending on I18n.locale. We are talking literally zero cost here. Mongoid even accounts for the reality of translations not always being present so you might want to fallback to another locale instead of not presenting anything. Wow..

So here is how to set this up with fallbacks inside your config/application.rb:

config.i18n.fallbacks = true
config.i18n.default_locale = :en

That's all, and here is how it works:

post = Post.new(:title => 'Hello World')

puts post.title # => "Hello World"
I18n.locale = :de
puts post.title # => "Hello World"

post.title = "Hallo Welt"
puts post.title # => "Hallo Welt"

I18n.locale = :en
puts post.title # => "Hello World"

Unfortunately this at the moment only works for regular fields, not for relations so I still have to do some work in my current project - but even the field i18n saves me a lot of headache.

Multiple fallbacks: Something that is not really covered in the documentation but becomes apparent when you look at the tests is that you can define a callback chain per locale. Meaning you can tell Mongoid to do go through a list of locales while searching for an existing value.

::I18n.fallbacks[:de] = [ :de, :en, :es ]

Mongoid will now first look for a german value, then fall back to the english one, and if that's not present either it will try the spanish one. Very handy especially if you run the locale schema down to country codes like 'de-AT', 'de-CH'. So for example this is the fallback chain for switzerland:

::I18n.fallbacks['de-CH'] = ['de-DE', 'de', 'en']
::I18n.fallbacks['fr-CH'] = ['fr-FR', 'fr', 'en']
::I18n.fallbacks['it-CH'] = ['it-IT', 'it', 'en']

Yes, Switzerland is a mess - but at least it gives you a good example for crazy fallbacks.

Filed under mongoid, ruby

New blog engine, this time my own

It's been some time since I did write a blog post, and I must say it was not for a lack of ideas. It was mostly the prospect of having to write these posts in HTML into my Wordpress interface that led me to abandon the idea.

And I think that's the main problem: I believe HTML is a bad format for writing posts. And no amount of WYSIWYG will change that, for all the subtle codehighlights I like in my posts the Wordpress WYSIWYG is just not the least bit useful. And don't get me started on posting source.

There are basically 2 solutions I had with my Wordpress installation: Either use a clientside syntax-highlighter or rely on Gist to post code in a human-readable fashion that's also easy on the eyes.

And I hated that.. I hated doing the back and forth between Gist and the site, and fixing errors in the code was always a pain. And as much as I like the syntax-highlighter script, it slows the page down a lot. Especially the front-page, even when the user may not be looking at a post with code at the moment code on the page will still be parsed.

A good solution to this problem seemed to be Jekyll, where you write your posts in simple markdown or textile and code gets syntax highlighted automatically on the server. I tried that some years ago, but for my style of writing it turned out to be even worse than Wordpress.

Why? Simple: I didn't start from scratch but rather imported my whole Blog into Jekyll when I started the experiment. So I was running Jekyll with syntax highlighting on on about 300 posts all the time. And the compilation did run for about a minute every damn time! To add insult to injury: at the time I didn't have a rsync capable hosting setup so I had to FTP the whole site.

So a few weeks ago I started to tinker around with Ember.js, and naturally I found myself writing a sample blog application. And that's when it hit me: writing a blog is f***ing easy. There is a reason any web frameworks worth it's salt is demoed using a blog engine.

And it turns out, what few features I want from my blog engine are usually all covered by these very simple demo apps :). So I ditched all the Ember.js experiments and started to write my own blog engine that had to do the following things:

  • Look exactly as my old blog
  • Accept Markdown input
  • Provide syntax highlighting
  • Let me schedule posts to appear at a later date
  • Preserve permalinks

And that's it - almost laughable how simple my requirements actually are. I decided to ditch some more complexity I felt no longer required (since RSS is dead) and built a really simple rails application ontop of the awesome libraries rouge and redcarpet. So my blog now accepts GitHub Flavored Markdown - all with less than 5 lines of code.

And the results are really awesome:

puts "I <3 Ruby"

So I sat down and built the blog in probably half a day. Maybe a bit more since I wanted to have some shiny like: nice tagging with autocompletion, real time post preview while typing and drag & drop file upload for images.

I'll post the code once I cleaned up the deployment mess I created (no secure database.yml management, secret.rb in git etc) but once done I'll post it here anyways.

Next up for me is trying to figure out a good way to make the blog theme-able. I'd love to experiment a bit with Rails engines and how this can be leveraged to separate themes from applications. And now that I've got my own little Rails playground once again it's the perfect opportunity to do so :).

Oh, and I need to leverage caching. After a cursory exploration today it seems caching would turn this blog into a statically generated static site. Only problem seems to be the cache invalidation, but I am sure I can devise some scheme to get around that that (short of doing 5 rm -rf calls).

Filed under rails, ruby, blog, site-news

Amazon S3 uploads fail in numerous ways for no apparent reason

Today I spent almost 30 minutes trying to debug a problem where one of our Servers was not correctly synchronizing files to Amazon S3.

First thing I tried was manually doing a s3cmd put of the files in question, and was immediately greeted by numerous connection reset by peer error messages. s3cmd is so smart, it even automatically retries for 5 times while throttling the upload bandwith - still to no avail. When down to 5 kb/s upload speed I started getting broken pipe error messages.

Well, obviously our server has more than 5 kb/s uplink so I suspected someone broke something on the network level, but everything else was working fine.

The final clue then came when I was running `s3cmd ls` on my bucket where I finally got a meaningful error message: the difference between the request time and the current time is too large. Huh? Yes! Doing a date on the server revealed that the server was almost 20 minutes behind the current time, thus the SSL connection to S3 could not be established.

Turns out we had not enabled ntp and so the server clock kept drifting for a couple of months. The solution was straightforward:

sudo ntpdate ntp.ubuntu.com
sudo apt-get install ntp

The first does an immediate update (only installing ntp will take some time to sync the clock back up, so doing a manual update fixes your problem immediately), and then it installs ntp to prevent this from happening in the future.

Filed under tools

Installing Eclipse on Mac OSX

Most Mac applications come bundled as a .app package that you simply have to drag&drop into your Applications for them to show up in Launchpad and Alfred. Or, if an application is a bit more complex you usually get a .dmg file that then installs like any normal MSI package you are used to from windows.

Not Eclipse for OSX. The Eclipse project maintainers do provide a OSX binary, but it doesn't look like most OSX .app packages in that the extracted binaries are a folder that then contains a .app.

If you only copy over the .app file you'll not be able to start Eclipse as it's just a starter script and the real Eclipse is living in the extracted Zip. I used to just start eclipse wherever I unpacked it, but apparently the right way to install it is to simply copy the whole extracted eclipse folder over to your Applications. Even though the .app is not in /Applications directly OSX picks it up and it shows up in Launchpad.

Filed under programmierung, java

When to use raw() and when to use .html_safe

What rails does very nicely is protect you from XSS attacks through aggressively HTML encoding anything you write between <%= %>. But there is one caveat: At times you may really want to render HTML from a string. So you need to tell rails not to escape your HTML in that case.

There are two methods of telling rails that a string is safe and should not be escaped: raw and .htmlsafe

And both do the same. They mark the string as safe (through the use of the html safe buffer) and rails will not encode it any more. The main difference between the two: nil.

If you are doing things like: "<img src='#{..}' />".html_safe .html_safe is totally fine as the string will never be nil, but if you are dealing with strings that may be nil .html_safe will break since there is no .html_safe method on the nil object. (For example if you are loading something from a config value or the database) In that case using raw(...) will just ignore the string instead of raising an exception.

As always with these things: raw and html_safe make it very easy to introduce XSS attack vectors into your application. So use them wisely and only on strings you are sure to be safe.

Filed under programmierung, rails, ruby

How to expire all active Sessions in Rails 3

I know this sounds like a very simple task to do, but since I just spent half an hour reading up on how Sessions in Rails work I decided it's time to put this up so I can Google it again next time :)

First off, you need to know how you are storing sessions in your application. Rails supports 3 types of session storage: Stored on Disk, in the Database or through Cookies in the client. You can check which one you are using in /config/initializers/session_store.rb.

Cookie based session storage

Rails 3 defaults to storing the session in the client using a session cookie. This means that the user_id along with all the data you put into the session hash is serialized into the cookie and sent to the client. It's also not encrypted, only BASE64 encoded so if you are storing anything sensitive in there you are doing it wrong.

But for simple things like the current user_id the cookie based session store is just fine and also a lot faster than the alternatives.

Expiring the cookie though is a bit more involved since you can't reach out to all clients and delete their cookies at once. But, and that's the important part for what I was doing: This cookie is signed with a SHA-512 digest using a secret key that is only present on the server. So the cookie cannot be tampered with on the client, and this is also your avenue of attack when trying to expire all cookies:

Simply change the secret that is used to sign the cookies. All previous cookies are invalidated as their digest is no longer valid.

Doing so is simple, first generate a new secret using rake:

$ rake secret 10dfec4781b682762a731a5e88af78521fc3e0f...

Then copy/paste that new secret into your config/initializers/secret_token.rb:

MyApp::Application.config.secret_token = '10dfec4781b682762a731a5e88...'

Once done deploy the application and all existing sessions are invalid at once.

Database backed session storage

If you are using the Database to store the session it's rather trivial to expire all existing sessions using rake:

rake db:sessions:clear

File based session storage

Like with the database simply run the following rake command:

rake tmp:sessions:clear

Hope this helps..

Filed under programmierung, rails, ruby

Optional locals in Rails partial templates

In Rails it is very advisable to not use any instance variables inside your partials if you want to re-use that partial in a different context.

It is much better to simply leverage locals that you pass into your render like this:

That way you can simply use posts inside the template without having to rely on @posts coming from the controller (and it's also not dependant on controller code any more but rather only on data inside the view that's calling it.. very handy).

But, as always there are times when you have additional locals that are optional to only some contexts. In my case I had Kaminari paginate the list I was passing to the partial, but in some other views the list was not be paginated. Turns out you can't do the usual rails style if foo here because the variable is simply not defined and Ruby will throw an error (unlike with @variables in Rails).

That's where the defined? method comes in very handy (as suggested by Thorsten Ball):

If you simply want to assign a default value to a local you can use the ||= operator:

This enables me to use the partial in several ways without having to re-introduce the locals at every corner:

Filed under programmierung, rails, ruby

My Photography business

Projects

dynamic css for .NET