Migrating to my own personal cloud
In the last decade, I have made quite a few migrations when it came to my personal files, email, calendar, contacts, music and video. I started out with having all my files and data stored locally on one computer. This data was synced to my mobile devices using various programs (ActiveSync, iTunes, Windows Briefcase, Palm Hotsync), which was rather messy. The problem was that I had to physically connect my mobile devices in order to sync them, and that the data on those devices would be out of sync between synchronization moments. It was also a rather tedious task to switch to a new computer, as I had to manually migrate all data (which was rather difficult for applications such as Outlook Express or iTunes).
The second phase was that I started using various online services. My email migrated to Hotmail, I synced my files using Dropbox and started using iCloud for my calendar, contacts and reminders. I’ve given this a second thought lately, and it is beginning to worry me. I am starting to lose grip on my data, it’s not my own anymore and the companies behind those services can decide to change their services drastically, without giving me any notice. What if copyrighted files will be automatically deleted from Dropbox in the future? What if Hotmail decides to disable access to your email using third-party applications, to lure users to their websites and watch their ads? What if the Hotmail spam filter starts to become too aggressive? What is one of the companies started charging money? What if a company goes bankrupt? What is my files accidentally get deleted?
A lot of what ifs, which is why I recently migrated all my data and files to my own personal cloud. A cloud that I control, where I make the rules and where I am not vulnerable to decision made by others. A cloud that uses open protocols and open standards with clients available for most devices on the market. So how did I do it?
I started out with an email address assigned by my ISP. Their mail-server used the POP protocol, which made it impossible to sync changes back to the server. It was only possible to download or remove messages from the server. Sorting mail in mailboxes, syncing read-statuses or storing sent messages were not (easily) possible. I changed to a Hotmail address later on, but again, it only had POP access with all the problems stated earlier. What bothered me most were the changes in how I could get access to my email. Hotmail added Exchange ActiveSync support in 2010, but I’d rather not use a proprietary connector to get my email.
That is why I bought a cheap hosting package that supports the IMAP protocol for email. IMAP is a great way to access mail: It actually syncs emails between the client server, instead of letting the client only download messages. Multiple clients can access the server simultaneously. Read-status can be synced back to the server, so that it’s available on all devices. Mail can selectively be cached for offline use. On my desktop, which has tons of storage and bandwidth, I cache my entire mailbox, on my mobile devices, I only cache the message title and content. And because it is open standard, it works on every device I own, even my outdated Ericsson R380 from the year 2000 and my Psion Revo from 1997.
In order to access my email using a browser I use Roundcube, a web-based IMAP client based on PHP and MySQL. It runs on the same hosting package mentioned earlier. Roundcube is a very smooth and stabile client that offers almost every function native desktop clients have. I usually access my mail using native applications, but this is a good fallback for when I’m using someone else’s device.
Migrating my email was rather easy. I had both my Hotmail account and my new email account setup in OS X’s Mail application, and I only had to drag the emails from the Hotmail inbox to the inbox of my new email account. I did have to replicate my directory structure by manually creating new mailboxes, but that was a rather easy task.
The only problem I have is that I can’t use IMAP IDLE (pushmail) on iOS. Sabahattin Gucukoglu wrote quiet a good post Deficiency,”) about this if you’re interested why.
I used to store notes as RTF files in a directory on my computer. That didn’t change much when I migrated to online services, although I did sync them with Dropbox.
I read a great post by Ben Jisa about using the drafts folder of your email to store notes. This folder is synced between all my devices that are set up for email. The Notes application that is pre-installed on OS X and iOS also uses this method to store notes, so I can use the Notes application instead of my email client.
Calendar, contacts and reminders
I used to store my calendar, contacts and reminders in Outlook. After a lot of frustration I finally transferred them to OS X’s iCal and Contacts application using a third-party application. Syncing this data to mobile devices was done using a mixture of Microsoft’s ActiveSync, iTunes, Palm Hotsync, SyncML (via Bluetooth) and a couple of third-party applications. All applications synced perfectly, but as mentioned earlier, I had the problem that my data would be out of sync until the next synchronization. It would also conflict if I would do a three-way sync and I had changed data differently on multiple devices, which made synchronization a tedious task. And because I used multiple mobile devices with multiple desktop operating systems, I had to use various applications to sync everything together.
After switching to iCloud about a year ago, I had solved most of the problems. iCloud stores your calendar, contacts and reminders on their servers and lets you sync your devices with it. Synchronization between Apple devices was flawless and only needed a single sign on. Synchronization between other devices was also rather easy, as Apple using the (open) CalDAV and CardDAV protocol to synchronize calendars and contacts. Instructions to set up your device can be found on a blog post at pwnmyi.com. CalDAV and CardDAV clients can be downloaded for almost every platform.
But this does worry me a bit. iCloud was succeeded by MobileMe, which was succeeded by .Mac, which was succeeded by iTools. As you can see, Apple really likes to overturn their products. And those CalDAV and CardDAV servers are undocumented features, which may vanish in the future when Apple wants you to lock in into their ecosystem. Which is why I moved to my own solution.
I used the same hosting package as my email to host ownCloud. OwnCloud is an open source software suite for creating a cloud solution. It has tons of features, but I use it mainly to sync my calendars, contacts and reminders. It uses the CalDAV and CardDAV protocols, and it also has a very smooth web-client which I use as a fallback for when I want to access my calendar from someone else’s device.
Migrating was almost as easy as migrating my email. OS X’s iCal application can easily export the entire calendar (including reminders) to a single iCalendar file. This same ICS file can then be imported to a CalDAV server like ownCloud, again using the iCal application. I could use the same method with the OS X’s Contacts applications using a vCard file.
Music and video
I always used to have music and video locally on my computer. I accessed them using Explorer and played them by browsing the directory structure and opening them in a media player like VLC or Windows Media Player. After switching to OS X, I started using iTunes to manage these. This meant that I didn’t have to worry about the directory structure, as iTunes took care of it. I could also use metadata to find files. iTunes also has some good synchronization tools, making it easy to sync to iPods, iPhones or other devices using third-party plugins.
Music and video’s are a bit different from the previous two items. They are not very personal, consume a lot of disk-space and don’t need a backup solution (I don’t mind if I lose them in a crash), which is why I used a slightly different solution. Currently, I am storing my music and video’s on a Synology server. I know that they are usually referred to as a NAS (Network Attached Storage), but given that it is basically a fully blown Linux server, naming it as server is more suited. This device is stationed in my home and is connected to the internet, so that I can access the files everywhere I want using every device I want. I am using a mixture of AFP, FTP, SMB and WebDAV to access my files. But this actually is a step backwards, because I lost the rich metadata and library functionality I had in iTunes. Synology offers a couple of easy to install packages for their NAS appliances, but I went with the third-party Plex - Plex Media Server. It scans your media directories, builds a library with metadata gathered from the internet and lets you browse it using one of the native applications (available for Windows, OS X, iOS, Android, Windows Phone, LG televisions, Samsung televisions and more) or the web-application. It even transcodes video on-the-fly to a lower bit rate for when your device or bandwidth can’t handle the original file. When using the premium version, it is even possible to sync the library to your local device for offline playback.
The only thing I dislike about Plex is that it is not open-source and that it doesn’t use an open standard for communication. They do supply an open API though. Because Plex is just a front-end, I don’t really care about this. My data is still easily accessible and still fully in my control.
This solution also has another problem. Because this Synology server is hosted in my home and uses consumer-grade hardware, it is not very reliable. I have had 200+ days of uptime on this device, but I wouldn’t trust it in a professional environment. Because I only use it for music and video, this is not a large problem, because I don’t mind if it needs an occasional reboot or if all my files are gone.
Documents and photos
Documents are still a bit difficult. I have a highly organized directory structure containing all my files on my local computer. I have synced some files with Dropbox, but not all files. The problem I am having is that I don’t want my sensitive files (financial and legal documents, taxes, license keys) on their servers. I could put all my files in a TrueCrypt container before I upload them, but that would be rather cumbersome. I also ran into the storage limit, as I have about 12 gigabyte of documents, and Dropbox charges a hefty monthly payment for extra storage.
OwnCloud, which I have installed on my hosting package, supports WebDAV and offers synchronization applications for all major desktop platforms (Windows, OS X, Linux). But I ran into the same problem as Dropbox, as my hosting provider charges a lot of money for storage and bandwidth.
Both solutions are mostly limited mainly by money. Because I only use these documents on my main workstation (their mainly related to finances, work, software development, vehicles and holidays), I don’t have any real advantage to hosting these documents in a personal cloud. This is why I’d rather not spend money on doing so.
The third solution would be to use my Synology server, which practically has unlimited bandwidth and very cheap storage. Synology offers a Cloud Station package, which also offers a synchronization application based on WebDAV. I did find this application to be a bit buggy, which made me worrisome as it did process some quite import documents. I could also install another instance of OwnCloud, which would work perfectly in theory. But there actually is another reason why I’d rather not use my Synology server, which is one I mentioned earlier. The server is hosted in my home and uses consumer-grade hardware, which is something I don’t rely on for hosting my important documents, even if they’re replicated on my workstation.
I also have my photos. I used to have a directory structure to contain my pictures, but after switching to OS X, I switched to using iPhoto. Like iTunes, this meant that I didn’t have to worry about the directory structure anymore, as iPhoto takes care of this. It has some very smooth features for managing, editing and publishing photos. But just like my documents, I currently only store it on my main workstation. Both Synology and OwnCloud have some very good photo management packages, but iPhoto still is the best application for photo management. It is also blazing fast, because everything is executed locally. I do sync them to my mobile devices using iTunes. iTunes automatically converts the photos to a lower resolution when doing so, in order to save space.
Bottom line, I only have my documents and photos on my main workstation. Storing them online costs a considerable amount of money, which is not worth it. Storing them on my Synology server is not such a good idea as it’s consumer-grade hardware hosted in my home.
As stated above, I used various methods to sync my data in a way that I have full control over my data. I only used non-proprietary and standardized open protocols. Data is only stored in places that I own myself. I do have some remarks:
Having data in multiple places is not an excuse to not backup data. I regularly backup my desktops, servers and even my mobile devices. This is highly advised in every situation, no mater how you store your data.
I am using a hosting package from a hosting provider. This means that I’m still not 100% in control of my data. I could have co-located a server or rented a VPS, but I’m not a system administrator, so I’d rather outsource that task to someone who knows what he’s doing. This is better solution when looking at security, uptime, data integrity, time and money.
Be sure to use HTTPS when using CalDAV, CardDAV or WebDAV. If you use HTTP without SSL, everything you sent or receive (including your password) can be intercepted using a basic network sniffer.
Yes, it was not as easy to configure as using online web-services. Yes, when I have to configure a new device, I have to configure my email, calendars, contacts, music and video separately. But I can now use every device I own, as clients for IMAP, CalDAV, CardDAV, WebDAV and Plex are available for all current devices.
I couldn’t help but notice how Apple, unlike I expected, uses a lot of open standards, protocols and file formats. Their iCal and Contacts applications on both OS X and iOS can easily be configured to be used with a server that uses the open CalDAV and CardDAV protocols, and the default export format for the OS X applications is the open iCalendar and vCard file format. iCloud apparently uses CalDAV and CardDAV under the hood and Apple has even published the open source Calendar and Contacts Server for Linux. To put things in perspective, take a look at the “Comparision of CalDAV and CardDAV implementations” page on Wikipedia. Apple’s own open source Calendar and Contacts Server has the most extensive support on both protocols, as do the built-in applications in OS X and iOS. Windows 8, Windows Phone 8 and even Android currently do not natively support CalDAV or CardDAV. This has really blown my mind.
While I was writing this blog post, Google axed a lot of features in a so-called ‘Winter cleaning’. These features included synchronization with older Nokia cellphones, creating events and checking your calendar via SMS and the cancellation of Google Sync using the Microsoft Exchange ActiveSync protocol. The timing of this decision couldn’t be better as it proofs exactly that the “What if?” questions I noted earlier can actually happen.