Drobo File Transporter – The Four Levels of Support Hell (or thou shalt not backup iMovie libraries to Drobo’s File Transporter)

Subsubtitle: When you write a backup software, make sure you understand the capabilities of the file system you’re copying from.

Since September 2013 I have been a more or less happy user of the Drobo File Transporter private cloud. The idea seemed and still seems perfect:

You buy two File Transporters and connect them to the internet in two different places. On each of your computers you get a „Dropbox-like“ folder that syncs to all of your computers and is even accessible from your iPhone or iPad. But its size is only limited by the size of hard disks you provide for no monthly fee. It’s a redundant backup system with the ease-of-use of Dropbox!

Alas, it’s not perfect. In the early days, the backup software was very CPU intensive but that got fixed eventually. Then followed a few months of honeymoon when the two Transporters just worked. And then, they suddenly started to run full. First the 500 Gb disk I had, then the 1 Tb disk. And I had only backed up 300 Gb.

I contacted the File Transporter support. (Don’t bother trying to activate your activate your Transporter account at https://support.filetransporter.com/activate. That never worked.) And I met support hell.

I was asked to perform the good old „Have you tried switching it off and on again?“ trick. That didn’t help. I had to fill out a long questionnaire about the kinds of disks I had connected, the File Transporter firmware version etc. I was asked to submit the full File Transporter log files.

By then, a full week had passed. The result: I was kindly informed that apparently my disks were full and I would need to switch them for bigger ones.

Obviously, I refused. The case got escalated. I was asked for screenshots of the management website and the Finder window that showed the amount of data I had backed up. I guess these established that I was speaking the truth here so … the case got escalated.

22 days after my original support request I receive an e-mail from 3rd level support. He asks me to send in the log files again. Something wasn’t right with the old ones apparently. At this point, my patience had run out, let alone my confidence that I was receiving support at all and not some form of online occupational therapy.

So I debugged the issue myself. If you still care, or come here via Google, this is it:

Do not backup iMovie 10 libraries to Drobo File Transporters.

As of Software/Firmware version 3.0.21 the iMovie library will fill your entire disk.

The explanation: The File Transporters format the disks with the ext2fs file system internally. I mounted them on my Mac and started to look for my backups.. I compared the size of the backups to the size of the original folders and quickly narrowed the problem down to the iMovie library.

The iMovie library folder contains a symbolic link „.fcpcache“ that links back to its parent, the iMovie library. I have no idea why it does that. It certainly seems unusual but it is just as certainly not forbidden to do that.

The File Transporter treated the symbolic link as a normal folder, and copied its contents and thus the whole iMovie library again. And again. And again ad infinitum or rather until the disk ran full.

iMovie will recreate the link if you delete it, so that’s not a workaround.

Tracking Election Promises with Scala and Play!

A few months ago I decided to learn Scala because it looked like an interesting, type-safe, concise and multiparadigmatic programming language. And all the cool kids were talking about it. At the same time, about two months before last year’s elections to the German parliament, my internet friend and journalist Jan Falk had the idea to build a German version of the Obameter tracking the campaign promises of the next German government.

Thus, the wahlversprechen project was born. A Scala and Play! based web application. Today, about 6 weeks after the new government formed, the complete coalition treaty signed by the two governing parties is online as are the campaign promises by both governing parties, the Social Democracts and the Christian Democrats.

We itemized the election programs as well as the coalition treaty and identified the promises that can be rated as kept or broken, i.e., the promises that are precise enough that failing to implement them is actually possible. The election promises are categorized based on the responsible Ministry, they are tagged and searchable.

Each election promise has its own page which shows how it has been rated (i.e. if the promise has been rated as a kept promise, a broken one, if it has stalled, or is in progress), the relevant quote from the election program and the editorial updates linking to press reports or even the passed law. To let interested users and readers contribute their opinions or even updates on the state of some promises, each election promise page has its own Disqus comment thread.

The start page shows at a glance how many promises have been kept, were broken, or ended in compromise:

Progress Bar on Start Page

To import long lists of itemized election promises the application is able to import Google Spreadsheets. Site editors can post editorial updates, change the tags or fix typos via the web interface of course (Note the Markdown syntax in the source of the quote):

Editing View for Election Promises

What’s next?

There are a couple of todo items pretty high on my list:

  • The site needs some (JSON) export to free the data we have assembled
  • All templates are in German at the moment and need some internationalization. (The code is in English of course as it should be)
  • I’ve already started work on a news aggregator that tries to identify the election promises the media are currently talking about through simple statistics. The code is on github and it is already working pretty well.

What can you do?

Well glad you asked!

  • Use it as a reference and tell me if it works!
  • Tell me and the team if anything important is missing.
  • Comment, help us do the research and follow the news.
  • Run your own instance and track other elections!
  • Fork it & improve it!

Most of all, tell me if you find it useful. That always makes my day.

Better Function Objects in C++11

At the 2013 Meeting C++ conference, Eric Niebler argued in his keynote that we should be using function objects more often in C++. If I recall correctly, his argument was that function objects don’t participate in ADL.

Function objects have another advantage. They are easier to use in std::bind. If you pass a function pointer or a pointer to a member function to std::bind et al, you have to specify the exact function overload. Given

template<class T> 
struct foo {
    int bar(int a, int b) const { ... }
    int bar(int a, int b) { ... }
};

the following statement creates a unary function that takes an int x as argument and calls the const overloaded member function f.bar(0, x) on that instance:

foo<int> f;
auto fn_bar = std::bind(
    static_cast< int (foo<int>::*)(int, int) const >(
        &foo<int>::bar
    ), 
    f, 0, std::placeholders::_1
);

Eric advocated writing function objects like this instead:

struct bar_ {
    template<typename Foo>
    auto operator()(Foo&& foo, int a, int b) const 
        -> decltype(foo.bar(a, b)) 
    {
        return foo.bar(a, b);
    }
} bar;

The above bind can now be written as:

auto fn_bar = std::bind(bar, f, 0, std::placeholders::_1);

Much better. But wouldn’t it be better still if we could create a unary function object by writing

auto fn_bar = bar(f, 0, std::placeholders::_1); // 1

or even — let’s go crazy — by writing this:

auto fn_bar = bar(f, 0); // 2

Since bar is a ternary function that is only called with two arguments, it turns into a unary function fn_bar. Calling fn_bar(1) expands to bar(f, 0, 1). That is called currying and is supported e.g. in Scala and other functional programming languages.

All we need for this to work is a wrapper around our struct bar_ that can detect at compile time if

  1. bar is called with arguments that contain a std::placeholder
  2. bar is called with too few arguments

In these cases the object returned from a call to bar is actually a std::bind object, i.e. a function object. The first style of self-binding function objects has been implemented by Eric as part of his (experimental) range library. I’ve implemented the second style of function objects that automatically curry when the number of arguments is too low.

Eric’s imple
mentation supports chained function objects, it seems, i.e. calling f(g(std::placeholders::_1), 1, 2) should become a unary function. I didn’t get to that.

While I like the idea of leaving out the placeholders too, my own approach suffers from a serious drawback at the moment. My solution is, like Eric’s, based on std::bind and that needs to be passed the desired number of placeholders. Therefore my function object wrapper needs to know how many placeholders to pass to std::bind when it is called with n arguments. That means my function wrapper needs to know the arity of the wrapped function. There is no way to determine the arity of struct bar_ at compile time. bar_ could overload operator() of course with different numbers of arguments, so there isn’t a defined single function arity to begin with.

The solution: Remove my dependency on std::bind.

To be continued