Since I am travelling a lot ― for work as well as pleasure ― I also tend to suffer from jet lag a lot. Trying to get on top of that problem, I was first of all trying to find a way to reliably monitor and measure it ― once an engineer, always an engineer…

So I downloaded and installed a number of sleep tracker apps to see how good they were, and found the result to be shocking. The algorithm of the two I have tried so far was deeply flawed in a very similar manner. Below are the results of my ‘research’….

Pillow

This app would be absolutely brilliant, if not for the fact that its algorithm is deeply flawed. Instead of comparing your actual sleep against your goal, the sleep score is calculated by comparing the time you have been asleep against the time you have been in bed.

So, if I come home after a late night’s work and go to bed at 2300 hrs, then have to get up at 0500 hrs in the morning to go back to work, I get a very high sleep score, because I was asleep virtually the entire time I was in bed. Was this a good sleep, though…? Definitely not, at least two hours too short.

On the other hand, if I have a long lie-in on a Sunday, so was in bed from — say — 2200 hrs waking up around 0700 hrs and then snoozing until 1000 hrs, I get a very poor sleep score, because the time I was in bed vastly exceeds the time I was actually asleep.

Funny enough, the second type of sleep which gets the very low score is by far the one I’d prefer to always have!

If you don’t believe me, carry out the following experiment: Stop your alarm clock when you wake up, and take a note of your sleep score. Then switch the alarm back on: Pillow will offer you to continue your sleep or start a new session — choose ‘continue’. When you next wake up, or if you just lie in bed for another half an hour, you will notice that your sleep score has drastically reduced. Why? You already had a good night’s sleep according to your first score, so anything after that is a bonus, right…?

Since the sleep score is unreliable, therefore all the other little gadgets — most notably your ability to create unlimited numbers of notes, which allow you to then analyse a very complex set of circumstances — are useless. If the sleep score has no relation to how good my sleep was, then the notes will recommend the wrong thing, because they are trying to raise my sleep score. So, at the moment, going by my sleep notes, I should be working late, eat late and too much, never ever have any exercise, and drink lots of alcohol, because those are the nights that get the highest score.

Also, with all its cleverness, it never seems to have occurred to the programmers to take into account the one variable the app has always access to, and which probably has the largest impact on sleep quality, namely the time I went to bed. Believe it or not, your bedtime is completely ignored in the sleep quality analysis.

Sleep Better

The next tracker I tried was Adidas-sponsored “Sleep Better”, mainly due to being recommended by the Apple Health app.  Unfortunately, I quite quickly figured out that it suffered from exactly the same problem as “Pillow“, see above, with exactly the same flaw in the sleep score algorithm.

Additionally, it has much less features.  For example, unlike the unlimited and fully customisable notes provided by Pillow, there are only six preset notes, namely

1. Worked out
2. Stressful Day
3. Not my Bed
4. Ate late
5. Caffeine, and
6. Alcohol

“Not my bed” seems like a good idea, and applies to me quite often ― travelling a lot on business ― but then again is useless because I almost never will  have a socket close enough to the hotel bed to actually run a sleep-tracker overnight.  So, this is a note that I personally don’t need, and if I am limited to six notes only, where I would really rather track something else ― well, tough luck, take it or leave it, it’s not customisable…

Having said that, Sleep Better has two feature I like very much:  Dream Notes, which track the quality and quantity of dreams, and Insights, which tracks sleep quality against moon phases, which I consider an interesting experiment.

Conclusions

I grudgingly have returned to using Pillow for the time being, as Sleep Better‘s very limited note capability just was not up to the level of analysis I would like to perform to see what influences my jet lag.  I have left feedback on Pillow‘s app page, which received only a very non-committal reply from the developers

“Pillow’s sleep quality score takes into account a lot of different criteria including the time to sleep, time asleep, time spent on each sleep stage, any existing audio data and many more.“

No kidding!  Well, hopefully the reply was deliberately vague in order to not to have to admit that the algorithm is faulty.  With a bit of luck, they will now take a good look at their algorithm, and reconsider what they are actually trying to track.

I will update this post as and when new information becomes avaiable…