Shoving Music Into a Cloud: My Life with Spotify

Shoving Music Into a Cloud: My Life with Spotify

I constantly have some sort of music playing. In the background, in the foreground, in another room, it doesn’t matter, I want something playing at all times. Considering my Last.FM account passed 80,000 scrobbles recently, I’d say I have some pretty high demands of my music.

In September of 2014, I decided to fully ditch my offline music library.

By that, I mean I purged the music on my phone, the music on all of my computers, and didn’t even bother making backups of those files. I went all in, in the name of a grand experiment: what would life be like when relying completely on online music streaming services?

Specifically, I went whole hog with Spotify. Which was great timing, because it had just come out in Canada.

TL;DR: It’s pretty great, and I only have minor setbacks once in a while.

Internet Always Required

One of my biggest reservations was that I constantly needed an internet connection to listen to music. Meaning if the internet were to go down for an extended period of time, or if I were out in the boonies with no reception, or if I’m commuting on the subway, I would have no music. I’ve been in those situations several times each, and frankly, there is no easy way out of it.

Spotify has a feature for Premium members to make their music available offline. I haven’t had the chance to try it, but I assume it will just be music downloaded to my devices, similar to filling my library.

A persistent internet connection will be the largest problem with making the full leap into online music streaming providers, and there’s no way around it. However, living in Toronto means I will have a persistent internet connection about 98% of the time, and so this is really only a once-in-a-while issue for myself.

The Latest and Greatest

Another reservation about making the switch is that the latest releases from artists may not make it into your platform of choice, for whatever reason.

As an example, when Taylor Swift’s 1989 album came out, I wanted to see what all the hubbub was about, so I typed in “Taylor Swift” into the search bar and found nothing. I mean, absolutely nothing. After some digging around, I learned that Taylor Swift and her record label didn’t want anyone on Spotify to listen to any of her music at all. End result? I didn’t give it a chance, at all.

As a counter-example, however, Drake recently released his new album, If You’re Reading This It’s Too Late. He released it during the evening, something like 11PM EDT, and I read about it in the morning. While it wasn’t attached to his artist page on Spotify yet, a search for the album name brought it up and let me listen to it within seconds.

So what’s the end result? Well, the latest albums and singles from artists won’t always be available, and there are alternatives to listen to them (cough, YouTube, cough) but it’s going to depend on the artist and the rightsholders.

For the most part, this will hardly be an issue. At least not on Spotify thus far.

The Oldest and Greatest

One of the benefits of having access to hundreds or thousands of artists and their full musical library is that you can feel like listening to something, and have it play within seconds.

Just this week, I felt like some old school hip hop on Monday, hard hitting trance music on Tuesday, lovey-dovey indie folk songs on Wednesday, indie electro on Thursday, and 90s R&B slow jams on Friday. Within seconds, I was able to find a playlist, an artist, or start a Spotify radio station and jam along to my incessant musical wants and needs for the day. No sweat.

Discoverability is Now a Thing

The lovely thing about finding random playlists or starting radio stations off of songs you like is that you sometimes run into artists that just resonate. It’s been happening non-stop since I made the switch, and I don’t think it would have happened as organically if I were listening to my offline collection.

As I write this post, I started with a featured playlist called “Indie Pop Chillout,” and I’ve already saved a few of the songs from artists that I have never heard from before. Call me a bit slow on the uptake with new music, but it has never been this easy to listen, discover, and save new music, going back to the days when I would record songs off of the radio using tapes.

Different Experiences by Device

Speaking only about Spotify, I’ve found that I get a different listening experience based on the devices I listen to. Because I’m not a Premium member, I will have ads playing between blocks of songs, and that’s fine. They’re unobtrusive, they can be informative, and the gentleman has a rather pleasant voice.

However, on my phone, I can’t pick and choose which songs to play. It’s forced shuffle play, apparently unless I have Premium. On my iPad and desktop PCs though, I can pick and choose whatever I like.

I’m not sure if that’s because a lot of people listen to Spotify on their phones and that’s the best way to encourage upgrades to Premium, but it can be a bit jarring at times. Like when I’ve shuffle play’d on a specific album, only to have it play things from different albums or different artists altogether.

I should probably take the plunge into Premium, but I really haven’t found a burning desire for it, and the annoyances are minor enough to ignore. I hope that doesn’t change any time soon.

Was it Worth The Switch?


Before I streamed entirely from Spotify (or any other music service,) I kept up a library of about 50-60 GB of music. I’d continuously feed it new releases and singles, and go through the work of maintaining the labels and metadata to make sure that I was looking and listening to the right thing. The library was constantly growing and would not fit into any phones or devices I owned in its entirety, so time was also spent picking and choosing what was the flavour du jour. And don’t even get me started on creating backups of this library.

Now, I don’t even think about these files that I used to collect and maintain. If I feel like listening to Stars tonight, I just type in Stars and away I go. Sure, it won’t always be available and I won’t always have the latest releases, but the accessibility and convenience sure are one tough offering to beat.

It’s pretty darn exciting to see what Netflix and Spotify have done to change my consumption habits with regard to movies, tv shows, and music. To think that these companies are creating such great offerings, that even the most fearsome pirates (arrrrgh!) have curbed their old behaviours.

I, for one, welcome our new cloud overlords.

Have you made the switch? What were your reservations before making the switch, and what did you end up experiencing? Let me know in the comments!

Patience, Planning, and Anticipation

I remember a few months ago, I was at an event where I was clearly one of the older people in the room, but still young enough to be able to relate, a student asked me: “If there was something you could tell to yourself when you had just started university, what would it be?”

I remember standing there, ruminating for a few minutes, and I came up with an answer.


Young Jon Lim had a penchant for wanting anything and everything to happen immediately. He wanted a few hours of work to create immediate results, he wanted things to be done now now now, and he wasn’t going to wait for a damned thing. He was kind of a dick, and I don’t really miss him. (Except his youthful ability to run for hours.)

If I were to tell the young me something to change, it would definitely be about being more patient. Certain things take time to percolate, changes will happen on a day to day basis, and it’s never worth panicking over a minute change in some small detail in your plan.

Which was another piece of advice I stumbled on while rambling about how impetuous I was in my youth: plans are fantastic guidelines and frameworks for getting things done, but do not treat them as absolutes.

I’ve made plans for years, and I’ve typically stuck to them, but sometimes a wrench gets thrown into the best laid plans of mice and men, and you really have to learn how to roll with it. Young me would cringe and figure out what the fallout was, focusing on the horrible things that were bound to happen, rather than figuring out what can be done to course correct or salvage. Why waste time dwelling on what’s gone wrong, when you can take control of the situation?

Lastly, I would encourage my younger self to curb my excitement for anticipation. I’m not entirely sure how to word that, but I would historically put the cart before the horse – anticipating the best case scenarios without having done any of the work. I would encourage myself to hope for the best and expect the worst, despite how cynical that sounds. It’s absolutely conservative in business terms, and it’s the best way to constantly be delighted and satisfied with everything else.

I’m not sure those students were entirely satisfied with my answers, but I told them that everyone’s journey is radically different, and the lessons we all take away from our respective paths will vary wildly.

What words of advice or life lessons would you tell your younger self?

Life Lessons From Reddit

When I surf through Reddit, I occasionally find threads that are just worthwhile to save, because they make me laugh, because they’re interesting, or because they cause me to think.

Tonight, I found one of those threads that really made me think, and you can find it here: What is something someone said that forever changed your way of thinking?

And I’ll leave it up to you to decide how it makes you feel, but I just wanted to share a few of my favourites…

I was 13 years old, trying to teach my 6 year old sister how to dive into a swimming pool from the side of the pool. It was taking quite a while as my sister was really nervous about it. We were at a big, public pool, and nearby there was a woman, about 75 years old, slowly swimming laps. Occasionally she would stop and watch us. Finally she swam over to us just when I was really putting the pressure on, trying to get my sister to try the dive, and my sister was shouting, “but I’m afraid!! I’m so afraid!!” The old woman looked at my sister, raised her fist defiantly in the air and said, “So be afraid! And then do it anyway!”

That was 35 years ago and I have never forgotten it. It was a revelation — it’s not about being unafraid. It’s about being afraid and doing it anyway.

My old boss, the CEO of a small hospital, told me a story from back when he was a lab technician (for simplicity, let’s call him Dan). Dan had forgotten to check some sort of mechanism on a piece of equipment he used, it malfunctioned and broke the equipment which ended up having around a $250,000 repair bill. The next day Dan’s boss called him in to talk about it, and he was sure he was going to be fired. His boss asked him why he didn’t do a proper check, made sure he understood what happened and sent him back to work. Dan asked him “Am I not getting fired? I was almost sure that’s what this was about.” His boss said “No way, I just spent $250,000 teaching you a lesson you’ll never forget. Why would I fire you now?”

It seems silly, but that attitude always resonated with me. Don’t make professional decisions based on emotional responses. Always know what your goal is when dealing with someone, and what exact problem you are trying to solve. Everyone makes mistakes, and yelling at them just makes them resent you and become defensive. Being calm and understanding will make people look up to you.

“Everyone you meet knows something you don’t.” My grandfather told me this, and it’s been a good reminder that I am surrounded by teachers.

“The person that you will spend the most time with in your life is yourself, so try to make yourself as interesting as possible.”

A small thing, but vastly more important than it sounds:

I was sitting in a bus once, and we came to the railroad tracks. There were some cars sitting inbetween us and the next red light, so if a train came, we’d be stuck until it had passed. That was always a couple of annoying minutes.

Then the light turned green, and the bus went across the train tracks without having to wait for a train. Pheew, crisis averted. Then, behind me, a mother said to her small child:
“That was too bad, we didn’t get to see the train today.”

That was the perfect way to frame that. Why not enjoy what you get?

Too many others to include, but a fantastic read.

Destiny: A Lengthy Review

Destiny: A Lengthy Review by Jon Lim

It has been exactly one week since the much-hyped Destiny was released into the world, and I wanted to really give it a fair shake before writing anything about it. At this point, I’m a level 17 Exo Warlock, currently on the starting point of Mars, have had my fair share of time in the Crucible, and I’ve participated in a handful of Strikes on the Earth and Moon.

If you understood none of that, it basically means I’ve played a good chunk of the single player game, I’ve participated in the multiplayer matches, but I have not had a chance to get into the end-game content, which would be Raids.

Pre-conceived Notions

To complete my disclosure, I played through a good amount of content on the public beta. I got right up to the level cap. I knew how to play the game the minute it was released on launch day, so I’m not sure if that changes this review. However, I will say I walked into the game with very low expectations, given that I was quite bored in the beta, and the whispers were telling me that not much would change between the beta and the launch.

Core Gameplay

Destiny by Bungie, Perfect Shooter Mechanics

Destiny, as a shooter at its core, is near perfect.

The controls are smooth and responsive, the guns feel impactful, and it’s extremely satisfying to blow up a huge chunk of enemies with a super. Enemies dodge, duck, dip, dive, and dodge whenever I’m around, and the variety of skills and weapons give you enough variety in build paths to really spice things up.

It’s a shooter with RPG elements, including level progression, but also weapon and armour progression. Both really add to the game, especially loot progression, but more on that later. The feeling of growth and progression are really well done in Destiny, and I’d like to continue playing to get a sense of what happens at the level cap and also what end game content feels like.

Graphics and Art

Destiny is absolutely beautiful.

Destiny is a beautiful game. Absolutely gorgeous graphics, really showing you what a next-gen title can do.

There’s something about zipping along the coastline on your speed-bike, as the sun is shining behind your character causing some lens flares, that makes you appreciate the care that went into building the world and making it look great. Crawling through alien tunnel systems or underground temples really made me want to explore the entire planet, and running into packs of enemies or treasure only sweetened the deal.

Seriously, the planets and areas that Bungie has created are absolutely stunning, and your first time touching down on a new planet always feels like an incredibly new experience.

Now, The Problems

Now that I’ve gotten my favourite parts of the game out of the way, we can start talking about the issues that are absolutely dogging this game. There’s quite a few complaints, so buckle in.

Your Shooter is Boring

The controls and shooter mechanics? Awesome. Fantastic. I would use them for every shooter going forward if I could. You took the best aspects of Halo’s controls and gameplay and made them better.

Everything wrapped around your shooter? Boring as hell.

Destiny can sometimes feel like a ghost town.

Every mission on each planet drops you into the same place, and you have to make your way to the progressively further objective. You’ll invariably fight through a few dozen of the planet’s enemies du jour, and you’ll reach your objective, where you deploy Ghost, who will tell you he will need time to hack into the system / steal data / mess with your enemies. But oh no, the jerks who we fought on the way in have discovered where we are and are sending a few soldiers to get you, whom are mandatory to kill before Ghost can complete his actions.

Save for a few missions, the variety of objectives is almost non-existent. You’re always going to have to fight your way to the objective, then you either deploy Ghost and defend him, or kill the big baddie that shows up.

Never mind the poor presentation of the story, the need to constantly go back to your ship, the long load times for going back to your ship or flying to a destination, and each planet being one large map that you will see over and over again through that planet’s missions.

The Community Sucks Because of You

Destiny's community could use some work for better communication.

I’ve run into plenty of people at the social hub, on planets during missions, and in multiplayer matches. However, there’s no easy way to communicate with any of them, except for doing the wave and dancing emoticons.

If you have no friends on PSN or XBL who play Destiny, you’re probably not going to have an easy time finding a Fireteam who you are able to effectively (voice) communicate with. There are ways around this, with subreddits like /r/fireteams, but that sort of defeats the purpose of being able to just jump in and play with the randoms that you’ll run into. Not that I’d want Destiny to become Warframe or anything, but being able to wave to people you run into and then being able to invite them to a fireteam, and being able to voice chat on that fireteam easily would be great additions.

A basic chat, easy fireteam creation, and fireteam chat. Any of those three would help facilitate the feeling of community and working for a common good that should exist in Destiny. The sense of community does not exist in Destiny simply because no one can easily communicate with each other, except when I express myself through dance.

Matchmaking and Multiplayer is … Odd

I don’t fully understand it yet, but I’ve played at least a dozen multiplayer matches, mostly on Control. If you aren’t familiar, Control is where you and your team control three points on the map and defend them from the other team who are trying to do the same.

Perhaps it was just my first couple of matches, but when I jumped on at level 6 (the minimum to play multiplayer!) it felt like I was getting curb stomped into the ground by much higher levelled people who had far better gear. The game says that level advantages are disabled, but I’m not entirely sure what that meant, or if that’s true. It just felt like I was getting wrecked by people who had better guns and armour, and better familiarity of the maps.

Skip forward to when I turned level 15 and I jumped back on Control maps: I was holding my own and often throwing down a 1.00+ K/D ratio, while usually leading my team in captures. I doubt I had become significantly better in my two additional hours of cumulative play time, and the only difference was that I had far better gear and skills for my Warlock.

It’s not a huge problem, but quite a horrible experience for anyone just dipping their toes into multiplayer, to get matched up with a bunch of level 20+ players who just splatter your face into the walls as often as they can.

Strikes are Awful, and Unsatisfying

Strikes are basically big bosses that exist as special missions on each planet, where you’re forced to be matched with other players to help you take on the challenge ahead. It helps to be over-levelled for these, trust me on that one.

Destiny's Strike missions are boring and unsatisfactory.

The notion of being grouped with two other people is actually pretty great, and I would have enjoyed it more had I been able to communicate with them at all!

Regardless, you’re once again dropped off on the same point, and you fight your way into the farthest part of the planet. Once there, you’ll run into a large boss of some sort that you’ll have to kill. It’s not particularly challenging, or engaging, or fun, to kill this boss. They are pretty much just a damage sponge with some sort of high-damage attack and/or minions to throw you off and give you ammo.

I played the Strikes on Earth and the Moon, and after both, I immediately said “Well, that was no fun.” out loud. They’re pretty cool bosses, as far as I am concerned, just far too predictable, boring, and completely unsatisfying wastes of time.

But There’s Potential

I didn’t want to end my review off on a negative note, because while Destiny has many, many, many issues that are present, I don’t doubt that Bungie will work the game into better shape given enough time. In addition, there’s a few things that have great potential and/or make the game enjoyable for me now.

Loot Drops Keep Me Going

Destiny's loot system is fantastic.

Perhaps just my love of ARPGs is propelling this forward, but I absolutely love the item system in Destiny. I love getting new items to compare side-by-side with my existing item in that slot, and min-maxing myself forward.

I enjoy watching my attack numbers spike as I make the right choices, as well as attacking faster, throwing grenades farther, or even the simple quick reload. I wish some of the cooler characteristics show up earlier on, to really draw you in, but it does force you to grind through some of the more boring parts of the game before you get to the fun guns and armour.

There’s an NPC in the Tower called the Cryptarch, and he levels up as you interact with him more and more and you receive items in your mailbox as well as discovering better items through Engrams, which is the Destiny equivalent of the “unidentified item.” Really enjoyable system, though I haven’t spent much time figuring out all of the benefits of levelling him up, nor have I received anything for levelling him up yet.

Multiplayer is Still Fun

Destiny's multiplayer leaves much to be desired, but is still fun.

I’m not huge on competitive multiplayer games, so I initially shied away from it in Destiny because I really just wanted to do some co-op missions and blast through alien jerks with friends by my side. However, once I tried multiplayer, I was pretty hooked. As simple as it can be, or even as unfair as it can be, you’re still able to kill people and contribute to your team’s success, and those actions are thrilling when you do pull them off.

On one control map, I single-handedly defended a point just by having well-placed shots and grenades, and anticipating room invasions well. It also helps that a Warlock’s super ability can clear a room quite easily, but finishing that map with that many kills and defends was a huge rush, and it made me keep going.

Playing With a Friend (or Two) Makes it Better

I had been playing solo until about level 9, because no friends of mine were ever online or even close to my level (they were usually further) so I never had the chance. However, one night, a buddy popped on and had a bit of time to kill, so we started up a PSN party, made a fireteam, and off we went on a few missions on the moon.

It was a far, far better experience playing with someone else doing the same missions as you. You’re able to tell each other enemy positions, you can chirp each other for missing shots, and you can have friendly competition on who can snipe the most guys from afar. It’s a completely different dynamic, being able to speak and verbalize as you play, compared to playing solo.

I mean, this is a “duh” moment, but it really shouldn’t need friends to make it more fun. However, given that it does, that’s a huge level of potential if Bungie were ever to fix fireteam creation and communication.

Public Events are Amazing

Destiny's Public Event system is incredible and fun.

Public Events are completely randomized events that occur on the planet that you’re on. I’m not sure if you have to be on the same mission as everyone else, but they appear and everyone in the area is alerted. You simply run in and start performing the objective (usually kill something or protect something) and you’re automatically included.

They’ve been tremendous fun because the entire area bands together to complete this event, and it’s always a good time seeing 5-15 other guardians shooting at one big baddie or protecting something important. I wish that’s how missions worked, or Strikes, or Raids, but I will settle for these fantastic and delightful random public events to get my fix of social and community interaction.

The Real Question: Is Destiny Worth Buying?

Initially, this would have been a resounding no. The game was boring from the outset, and repetitive to a fault, despite great shooter mechanics and an absolutely beautiful game. There are a myriad of issues with regards to being social, the lack of mission variety, and boss fights being just a lengthy session of “press the right trigger.”

However, Destiny has started growing on me. I’ve enjoyed the variety of planetary settings (despite having to run through the same set pieces for every mission,) the multiplayer can be thrilling and addicting, the loot system is pretty compelling, and public events are delightful.

If I had to give you a definite answer, I’d say: yes, Destiny is worth buying. But I’d probably wait until it went on sale a few months down the line, or pick it up used. It’s a fantastic shooter at the core, with tons of room for improvement. Hopefully, down the line, improvements do come because it’s an enjoyable game, but definitely not for everyone.

Destiny has been plenty of fun.

Sept 16, 2014, 8:57PM EDT – UPDATE: After watching a bunch of people try to take on the Vault of Glass, the newly released Raid, I retract any desire I had for matchmaking for Raids. It looks impossibly hard even with a well coordinated group of people, and doing that with a bunch of strangers would probably be an exercise in frustration. Looks fun though!

Google Spreadsheets: COUNT with Multiple Values in Cells

It’s been more than two years since I wrote a post on using Google Spreadsheets to use COUNT and FILTER in tandem to count values with multiple conditions.

Circling back, it does very well at allowing you to count when you need to validate multiple conditions. However, it falls apart the minute that your data is either messy or comes grouped with other data inside of a cell. Here’s an example where my original post would have trouble with:

Google Spreadsheets: Multiple Values in Cells

Why does it fall apart? Each cell can have multiple values inside of it.

To dig a little further, we have to understand that my original post called for using the FILTER function to pare down the data set into the specific rows that we need, and then using the COUNT function to count the result of the filter. However, FILTER does not allow for wildcards in its conditions for filtering, making it extremely difficult to account for data that is in the same cell as other data.

Luckily, there’s an easy way to get around it, using a crude but workable solution for counting values with multiple conditions, with sources that have multiple values in cells, extensible to messy or not-very-nice data.


The Scenario

In the above example, let’s say we’d like to count the number of items inside of Column A and Column B for every row that Item A appears in.

Manually counting them, we know that for Item A, it is listed on a total of 3 rows (Rows 2, 3, and 7) and has 7 values listed to its name in Column A, and 4 values listed in Column B. Hopefully I haven’t lost you, but I’m saying that we count the number of values inside Column A every time Item A is listed, and we do the same for Column B.

How would we go about doing this in Google Spreadsheets?

For the impatient, here’s what the formula looks like:


I know, I know, it looks odd and confusing, so let’s break it down starting from the very inside formulas.

=SEARCH("*A*", A:A)

This is what we’re using as a condition inside of FILTER. SEARCH allows us to parse through the values in a column or row and find characters or values that we’re looking for. In this particular instance, I am looking for the character “A”, but you could be looking for whole words, sequences of characters, you name it, and you have the possibility of adding wildcards on the outside of the term you’re looking for to make it catch every instance properly. Experiment with this, because I’m not 100% sure on whether or not the wildcards help in every situation.


When SEARCH is used in conjunction with FILTER, it returns a list of rows or columns (as specified by the first parameter in FILTER, in this case it’s B:B) that returned a TRUE condition for the SEARCH. The result would look like this:

X, Y, Z
X, Y, Z

This is returning B2, B3, and B7 respectively from my above example, because they had Item A in their Item column.

=JOIN(",", (FILTER(B:B, SEARCH("*A*", A:A)))

Using JOIN, you can combine all of our FILTER results into a single cell, where each cell’s values are separated by a comma (first parameter from JOIN, in this case “,” means separate values using a comma) and makes it nice and simple for us to analyze just this one cell.

X, Y, Z,X, Y, Z,Z

This is what our data now looks like, all inside of a single cell.


We arrive back at the original Google Spreadsheets formula, with SPLIT and COUNTA appearing. SPLIT allows us to break apart a single cell into many cells that contain each and every value that we have in the cell, as long as we let it know how the data is separated. In this case, it’s separated by commas because of the work we did above with JOIN, and we use COUNTA to count the non-numeric values that the data represents.

Doing this for Item A, we can quickly see that it has 7 items in Column A, and 4 items in Column B.

Easy peasy eh?

I linked the functions into the documentation for Google Spreadsheets as I listed them above, but here they are again, just in case:

If you end up trying this out, let me know what you think, or if you have any questions, or if you have any improvements or suggestions!

Scraping Steam for Data using Python + BeautifulSoup

As promised in yesterday’s blog post about analyzing public Steam numbers, here are the juicy technical details behind scraping a website using Python, and a Python library called BeautifulSoup.

Python Logo

The Method

I chose to use Python because I’ve been using it for a little under two years to do number crunching, as well as building a few automation scripts for work. It’s very lightweight, very easy to read, and quite a mature language.

That said, you can probably do this in whatever you feel like, but my approach consisted of the following steps:

  1. Poke the Steam & Game Stats page and get the HTML page that is served up to the browser
  2. Parse the HTML code and pull out specific numbers that would be useful for analysis
  3. Open a specified CSV file, and add lines to the file with all of the relevant data
  4. Close file, standby for next script run

What Was Used

The script is a very small file (33 lines!), and uses the following:

And you can take a look at the Gist itself to see the full script, but I am going to use this post to explain some of the methodology behind the script, to help people who want to learn about writing in Python and scraping web pages!

Alright, shut up, explain your code.

Of course!

steampage = BeautifulSoup(urllib.urlopen('').read())

This gets the ball rolling for the scraper. We use urllib to open a connection to the Steam & Game Stats page, and then read it with the BeautifulSoup library. If you are unfamiliar, I know I was, BeautifulSoup is a very powerful Python library that makes it super easy to navigate, search, and modify the parsed code you receive from websites.

In short: read the code of a webpage using BeautifulSoup, and you get all kinds of methods to chop and screw it to your liking.

timestamp = time.time()
currentTime = datetime.fromtimestamp(timestamp).strftime('%Y-%m-%d %H:%M:%S')

I wanted to use a consistent timestamp when recording data into the CSVs because it would allow me to group results in a sane manner. It uses the current time (of the script running) and formats it into YYYY-MM-DD HH:MM:SS so that when imported into Google Sheets, it would preserve the actual date and time aspects.

top100CSV = open('SteamTop100byTime.csv', 'a')

You’ll see two open(…) lines in my script, and both of them point to a specific CSV. This is where I dumped all of my data. The second parameter (‘a’) made sure I was opening and adding to the CSV, rather than continuously overwriting.

for row in steampage('tr', {'class': 'player_count_row'}):
    steamAppID = row.a.get('href').split("/")[4]
    steamGameName = row.a.get_text().encode('utf-8')
    currentConcurrent = row.find_all('span')[0].get_text()
    maxConcurrent = row.find_all('span')[1].get_text()
    top100CSV.write('{0},{1},"{2}","{3}","{4}"\n'.format(currentTime, steamAppID, steamGameName, currentConcurrent, maxConcurrent))

This simple looking for loop pulls out the Steam ID for the game, the English game name (as listed on the Top 100 list), the number of concurrent players (as of the script reading the page), and the peak concurrent seen throughout the day (I forget why I wanted this.) It also adds that information on a new line inside of the CSV.

In addition, this loop shows you the simplicity of the power behind BeautifulSoup. Let me break it down into smaller pieces because each one uses different BeautifulSoup methods.

for row in steampage('tr', {'class': 'player_count_row'}):

When I dug through the Steam & Game Stats source code, I realized that every game was listed inside of a table row with the class player_count_row. Upon seeing the pattern, I simply asked BeautifulSoup to iterate through every single block or table rows using that class, and as they are all uniform in their markup, can consistently pull out the information we need.

    steamAppID = row.a.get('href').split("/")[4]

With BeautifulSoup, you can make direct references to markup (as seen above), and then grabbing parameters within the markup itself (like ‘href’.) I did this to grab the URL of each Steam game, break it apart based on where the forward slashes (‘/’) were, and pulled out the app ID that was nestled inside of the URL.

    steamGameName = row.a.get_text().encode('utf-8')

Like the above markup parameter grabbing, get_text() is a very neat function in BeautifulSoup that allows you to grab the text for a link. The Steam & Game Stats page uses the game name itself as the link text, so it was a breeze to add to our collection of data.

    currentConcurrent = row.find_all('span')[0].get_text()

Nabbing the current and peak concurrent users is the same procedure, so I only need to explain it once. The find_all() function from BeautifulSoup finds the markup that is specified. It takes every single instance of that markup found and creates an array that can be referenced for easy modification or evaluation.

With the same methods and functions, I managed to easily find the current and peak concurrent users for Steam altogether.

Simple, right?

What else?

There’s nothing else, really! Those 33 lines were more than enough to collect lines upon lines of data inside of two CSV files.

There’s plenty of work that went into the analysis of that data, but that’s for another day.

Thanks for reading! As always, happy to answer questions or take feedback, so leave comments or yell at me on Twitter!

An Analysis of Activity on Steam

I have wanted to flex my analysis muscles for quite some time and thought I’d get some practice on publicly available data: the Steam & Game Stats page.

Steam Logo

For the unaware: Steam is one of the largest gaming platforms available for PCs, and they have a wonderful stats page that lists the current number of concurrent users for each game, as well as the current number of concurrent users in general. I’ve always wondered what actual activity levels were, ever since seeing this page, and so I decided to scrape that page for a week or two and see what sort of data I could get.

NOTE: If you’re interested in the technical details behind the scraping, I will provide that in a separate post in the near future.

The data I have managed to mine from this page is trivial at best, but I had a ton of fun learning how to build the scraper (uses Python, BeautifulSoup, and a handy dandy cronjob) as well as figuring out the required Google Sheets equations to put it all together.

Some Fun Facts About Steam Activity

On average, 19.95% of concurrent players on Steam are playing games.

Here is a visualization of what Steam activity levels have looked like from March 7, 2014 to March 19, 2014:

On average, 1/5th of the Steam “concurrent” user base is actually playing a game. However, I will concede that this is not dead-on accurate, as I only have numbers for top 100 games at any given moment, but is reliable for estimations because games in the top 90-100 levels account for 0.02% of the user base, meaning any additional users playing non-top-100 games will be relatively insignificant in their effect.

There was a big spike, on March 9, 2014 at 4:00pm EDT (16:00) when the number of Counter-Strike: Global Offensive players ballooned up to a staggering 111,893 players. Presumably for the start of a tournament. (Haven’t dug into this one too much.)

Dota 2 dominates games played on Steam, accounting for an average of 5.58% of the concurrent player base.

Not a real surprise, given the popularity of Dota 2, but the second most played game is Counter-Strike: Global Offensive, sitting at a distant 1.66% average, about three times smaller than Dota 2. This translates to an average of about 400,000 concurrent players, with the highest I’ve recorded at 673,018 concurrent players on March 15, 2014 at 10:00am EDT.

In fact, most of the highest numbers of concurrent players in Dota happens in the mornings from 9:00am to 11:00am.

However, might be indicative of the real struggle for MOBAs to fight against the titan amongst gods, League of Legends, which boasted an impressive peak of 7.5 million concurrent users in January 2014.

Hard to gain ground on such an entrenched competitor, but they’re definitely doing their best.

160 games have been a part of the Steam Top 100 between March 7, 2014 and March 19, 2014

While that sounds like a lot, we have to remember that the Steam catalogue currently sits at over 3,000 titles and growing, it’s pretty safe to say that breaking into the top 100 is no easy feat.

Further analysis that I could do as time goes on is to get a breakdown of the genres being represented in the Top 100 list, which would also provide a decent idea of what is and what is not popular on Steam. Not to mention that this is an extremely small sample size, it would be more worthwhile to get this data over a period of a year to make it really meaningful.

What’s Next?

Well, this is a big pile of data, and it’s growing by the hour. This is great, but what can I really do with this data?

For starters, the original goal was to figure out if there was a link between the digital marketing behaviours of publishers and the level of concurrent players on Steam, as well as growth or decline in player base from that activity (or lack thereof.) I will have to explore whether or not that is still possible to figure out, as there are a lot of marketing activities that are either harder to track down or even attribute towards the success of a game.

Secondly, I will have to step up the data storage game a bit to make it much more accessible. Currently, a Python script scrapes the Steam & Game stats page, adding a line to two separate CSV files with all the relevant data. I’d like to transition this into an actual database (probably MySQL) and maybe make it open to the public to poke at and do their own analysis.

Lastly, I’m really not sure. It was a fun side project in the first place, and I feel like it was a great learning experience and a fantastic way to brush up on my analytical skills.

Have any ideas or want access to the data? Give me a shout, I’m happy to share!

2013: Year in Review


2013 has definitely been my year: I managed to get into my dream career, I went on the trip of a lifetime through four European countries, and I finally flew the coop and moved out on my own.

Despite that, I have spent the last couple of days (when writing this, in late 2013) feeling like I was in a bit of a funk and thinking that I was actually disappointed in myself. However, I am pretty sure that’s incorrect, and that I am quite satisfied with what I have accomplished in 2013, but I wish to do much more for 2014.

As with 2012, I have created The Good, The Bad, and The Ugly for 2013.

The Good

I started 2013 off with a bang: I managed to get my dream job of building video games. I joined Massive Damage in early January as an Associate Video Game Producer, and got bumped to a fully fledged Video Game Producer in July. I work with a team of incredibly smart, talented, and passionate people who know how to have fun, and I’ve been a part of a bunch of products that have been incredible learning experiences.

Prior to joining, I had been a bit aimless in my career aspirations, but getting into the games industry has made the end goal clear – it’s always going to be about video games.

In June 2013, I went away for three weeks on a trip through Europe with three of my friends. We backpacked our way through Amsterdam, Paris, Barcelona, Ibiza, Madrid, and Lisbon, eating and drinking our fill for three weeks. It was a trip of a lifetime, giving myself a much needed break and exposure to the world at large, it also gave me incredible stories and experiences that I can never forget. I will eventually get around to actually working on the photos I took throughout.

Thanks to work, I became completely enamoured with board games, starting my collection with Zombicide. Since then, I’ve grown my board game collection to 11 titles, and I’ve spent many a quiet afternoon or evening rolling dice or throwing down cards. My favourite for the year is a definite tie between Zombicide and Ascension, and I look forward to discovering more titles in the new year.

In November, I managed to finally fly the coop and moved into an apartment in downtown Toronto with my best friend. It’s been a move that’s been in the works since July, but took its sweet time in being realized. The independence has been refreshing, the shorter commute has been a blessing, and a place to call my own has been pivotal in furthering the next steps of my plan.

The Bad

To supplement The Good, I have created a list of things that did not go according to plan this year, but in a change of pace, I am writing down how I think I will correct that for the 2014.

I haven’t saved nearly as much as I wanted to. For reasons I would rather not go into, my savings have stagnated in 2013, and I can really only blame myself. The part of me that demands fiscal responsibility has really slacked in the latter half of the year, and that will change. I am creating a few side projects that will hopefully alleviate the situation, and I plan to do a bit more nights-and-weekends consulting. I have many plans for 2014, and I’m not letting myself ruin it.

I did not get enough exercise at all this year. After going to the gym on an almost daily basis last year, I hardly went this year at all, to the point of stopping in July when things got a little too crazy at work for me to go in the mornings. I no longer have an excuse with my apartment having a pool and a solid gym, I will break this bad habit in 2014 or I will die trying.

I find solace in distractions. Every time I sit down to start a project, or improve myself, I find something to get lost in. Surfing the web (read: Reddit) has really crippled some of my ability to be productive, so I will change this in 2014, with specific time set aside to do only personal improvement work, no distractions allowed.

I did not complete any personal projects in 2013. I often came home exhausted, in a bad mood, wanting to do something else, or any combination of the three. This led to zero personal projects being completed in the year, and I feel like I let myself down. But, bad feelings aside, I will complete at least 1 in the first two months of 2014, or I will take away my discretionary spending budget.

The Ugly

I lacked any sort of organizational ability. I spent most of this year, outside of work, in a frazzled state of disorganization. I can chalk it up to any number of things, but once again, this comes down to my own silliness.

I think 2013 is the year I learned that I constantly overextend myself, and that I don’t spend enough time on any single project or task. I will become more single-minded and focused for 2014, as I don’t have the time or money to really do otherwise.

What’s Next for 2014?

I have high hopes for 2014, in that I’d like to finally grow and improve in ways that I haven’t before. I want to become a better producer, I want to become a better developer, and I’d like to be a far more complete adult. (Sounds silly, but it is exactly what I need.)

I am going to get back into regular exercise, I will set aside time for honing my skills and building new ones, and I will spend far more time developing and polishing ideas.

Thank you for taking the time out of your day to reading my year in review, I appreciate it. Feel free to get in touch if you’d like to give me feedback, ask questions, or share your own insights and experiences. I am always open to good conversation.

I am looking forward to 2014, and I hope you are too.

Happy New Year friends and family, here’s to you and yours.

October French Month: Week 1 Progress Update

Jon Lim - Duolingo Week 1

Original Post: October Project: Back to Basics with French!

After spending 15 minutes every day on Duolingo, every day for a week, I seem to be making solid progress. Level 5 in French, completed Basics, Basics 2, Phrases, Food, Animals, and working on Adjectives and onward.

The most difficult part has been setting aside time every night to learn, with the necessary quiet and state of mind to be receptive to what I learn. However, it seems like I’ve settled on a nice rhythm so far and haven’t missed a day yet.

You can take a look at my progress (and my stream!) in French on my Duolingo profile.

Here we go, week 2.

October Project: Back to Basics with French!

October Project: French with Duolingo

My September project failed miserably – I hardly had time to spend thinking about structuring an entire website let alone having the brain power to do so. I can’t blame anyone but myself but I’m course correcting and picking projects that are much smaller in scope and therefore much easier to keep up with!

October Project: Spend 15 minutes every day learning French on Duolingo.

My time in Paris, albeit short, was very fruitful. My French slowly came back to me and, despite trouble with vocabulary, I managed to hold my own while out and about. I’d like to brush up on my skills and get to the point where I can have decent conversations in French.

I’ll try to make weekly updates with a new screenshot of my progress, but otherwise, you can look at my sad, sad progress on my Duolingo profile.