Sunday, 31 March 2013

MongoDB Hackathon Sydney - March '13

A bunch of dedicated nerds assembled at the epicentre of Sydney's startup scene 'Fishburners' for a weekend of hacking on our MongoDB projects.

We're super grateful that we had access to 10gen's knowledgeable engineering team on a warm sunny Autumn weekend, when they could have spent the weekend at the beach enjoying the sunshine. Nothing beats having a go-to 10gen engineer to bounce ideas off once you've googled the hell out of the Internet.

Over the course of the weekend, Herb and I have made huge gains in our understanding of core Mongo concepts as well as getting feedback on design patterns and schema design, this is a wrap up of what we've learnt (remember these are all specific to our project News Maven: http://newsmaven.co and may not apply to your project per se).

Our app in essence is an RSS aggregator and reader. We have a Mongoose model for the following: Blog, Post, User, Folder which in MongoDB maps into the following collections: blogs, posts, users, folders.

We have a worker thread, which at a specified time interval, goes and grabs the RSS feed for each blog in the system and updates our posts collection with new articles (aka 'posts').

Whilst the actual idea is rather novel, there are some interesting issues that crop up for which MongoDB really lent itself to. Consider how to store the read/unread status of a blog on a per user basis.

There are three potential designs we could consider using:

Design one: Store read state per user, per blog subscription

In this scenario we would have a model that looks like this:

var ReadSchema = new Schema({
  userId: { type: ObjectId, ref: 'User' },
  blogId: { type: ObjectId, ref: 'Blog' },
  read: [{ type: ObjectId, ref: 'Post' }]
});

The way this would work is that each time a user reads a blog post a HTTP POST request is made to the server to indicate that a particular post has been read.

Pros:
  • Really simple to insert (it would just be a FIFO) 
  • If some thing gets marked as unread again, no problems we just push it on to the top of the FIFO stack
Cons:
  • Unread count needs to be computed on the difference between what's been read and what has still got to be read, and this would have to be done for each user, and each blog that they subscribe to - not an ideal scenario.
The pseudocode would be something like this:

function getUnread() {
  get ObjectIds for the Blog's last 5000 articles;
  For each ObjectId not in the Read Array, add to Unread Array;
  return Unread Array and Unread.count();
}

This has to be computed for each blog the user subscribes to; its two DB reads per blog (one to fetch 5000 articles, then another to fetch the current Read Array). If a user on average has 30 blog subscriptions, thats 60 DB calls per user

Scenario two 
Model: Unread
{
userId: foreign key to a user UUID,
blogId: foreign key to a blog UUID,
unread: Array of unread items
}

In the above scenario, when a feed gets updated, we go to each user and update their unread Array.

Pros:
  • Flat model
Cons: 
  • HUGE performance penalty (for each blog a user subscribes to, for each new article, we go and update the Array of unread items).
  • Disk cost is maximum, for each blog a user subs to, we store an Array of 1000+ ObjectIds, which will kill our storage costs.

I've run out of time, but I will definitely cover the rest of this in a follow up blog post! Stay tuned.

Monday, 28 January 2013

Deploying a single instance of node.js with supervisord

Nothing beats a technical blog post than some code. So here's what we're using to run node for our project Contactable.

[program:contactable]
environment = NODE_ENV=production
command = node app.js
autostart = true
autorestart = true

[supervisord]
logfile = /tmp/supervisord.log
loglevel = debug
pidfile = /tmp/supervisord.pid
directory = /home/app/contactable

[supervisorctl]
serverurl = unix:///tmp/supervisor.sock

[unix_http_server]
file = /tmp/supervisor.sock

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface


As you can see, the actual code is pretty straightforward, the options are explained to great depth on the actual supervisord web-site, but below is a quick breakdown.

To begin with, the 'program' part is perhaps the most interesting here, essentially, we are running node.js in PRODUCTION mode using the environment directive.

environment = NODE_ENV=production

This indicates to node that we are running the app in production and to therefore disable certain debugging and pretty printing error messages and general hardening. If you are using express, it would typically be tied in with something like this:

app.configure('production', function() {
  console.log('Running in production mode.');
  app.disable('x-powered-by');
  app.use(express.errorHandler({ dumpExceptions : false, showStack : false }));
  dbUrl = config.DATABASE_URL_PRODUCTION;
});


The second most important thing is that we have autorestart enabled - so that in the event node exits because of a crash in our code, supervisord will respawn the process (we'll leave error handling for another post).

Tuesday, 15 January 2013

Using RESTClient to test REST APIs

A tool I've become particularly fond of recently is the RESTClient (a Firefox add-on) by Chao Zhou.

One thing that trips a few people though, is getting RESTClient to test HTTP POST requests.

Most people get caught out because they forget to set the "Content-Type" header; since HTTP POST requests are generally of the type "application/x-www-form-urlencoded", you need to manually add this. If you fail to add this header, the server-side will not know that FORM data is being POST'ed and drop all of your FORM data.

So, here are a few quick and dirty screenshots that demonstrates how to set this up (hopefully this avoids any confusion!).

Create a new custom header

Add the header type & value


Test the header, value and body against your target API


Sunday, 13 January 2013

Introducing Vim Guru (a color scheme)

Since starting to use VIM, I've grown quite fond of a number of color schemes that are great on the eyes. Unfortunately, a large number of the color schemes I like, are built for Ruby developers, leaving much to be desired for JavaScript developers.

Taking Jo Vermeulen's textmate port of "Vibrant Ink", I've added better support for JavaScript developers, along with a few small color palette changes to aide visibility while editing.

Feel free to fork and edit vim-guru via github to your hearts content.
Vim Guru (a color scheme)
Vim Guru (seen here with no background transparency and using Adobe's Source Code Pro font).

Thursday, 27 September 2012

In the mobile app world, is field testing > unit testing?

All the bugs are in.

My first app Smooth Talker has been live for a few days and has received great feedback so far - but what surprised me the most is the variety of support requests we've had.

The different behaviours exhibited on the same type of hardware device has led me to some interesting learning.

i.e. No matter how much you test in a simulator, in a lab or how many test cases you write - nothing beats real world field testing.

You're probably thinking "duh of course"! But, taking a step back though - field testing is still pretty expensive for an indie developer - even with tools such as Test Flight.

Also, after doing some research, its clear that it is not common to do TDD for the mobile platform. This blurb I found by Pivotal Labs highlights the issue:
"In the mobile world, very few shops take TDD/BDD seriously. We can’t live without it. If you want to know if your developer is serious about doing Agile, ask them how they test. Even dyed-in-the-wool RSpec addicts will sheepishly say they don't test their iOS apps, other than the 'build, try and pray' method."
Which leads me to my current dilemma, because I have mixed feelings about test driven development for mobile applications. For a small dev house, the cost of doing TDD and extensive field testing is still too high - development times blow out if one half of the pair spends their time writing unit tests & it's particularly painful since we'll have to conduct lots & lots of field testing anyway.

So why continue to spend time writing unit test cases?

In the end, it might just be my lack of maturity & experience with Objective-C or the App Store process, that's really slowing me down? Maybe it's in fact quite cheap and easy to do both TDD and loads of field testing? If that's the case, I haven't quite figured out how to reach that balance yet...

Wednesday, 26 September 2012

Smartphones are going to make business cards obsolete

Once upon a time, when you meet someone in a professional context, they pulled out a business card with their details on it and handed it to you.

You would have then stuck it into your pocket or wallet and when it was time to return to the office or home, you would have put that card into your rolodex - only to shortly forget where & how you met that person or, under what context that person was important to you.

Of course, if that person was immediately valuable - you would take the time to scribe that person's contact details into Outlook or other contact management system. And if you were equipped with a smartphone, you might have used something like Cardmunch to take a picture of the business card and have someone manually transcribe those details for a fee.

But all current methods fall over because they all require repeated manual effort on the recipients part.

When Herbert and I were thinking through the problems faced by people using business cards, we knew we could build a better system. So over the next few months, we devoting our efforts to building an app that provides the least resistance to sending and receiving contact information - bar none.

Contactable will be the fastest & easiest way to provide someone your contact details using a smart phone.

So, if you deal with a lot of people & currently use business cards, I'd love for you to sign up to our beta list! Of course, we'll repay our gratitude by making you a kick ass app!

Be Contactable: http://becontactable.com

Monday, 24 September 2012

Smooth Talker - now available from the Apple app store!

After a shorter than expected wait from Apple (considering their increased workload with iOS6 and the new iPhone 5) our app "Smooth Talker" was released to the public.

It's the tool of choice for speech pathologists in the English speaking world when performing stuttering assessments. Calculating a percent SS (%SS) score has never been easier!

If you are a speech & language pathologist, or a parent of a child that stutters - grab the app from the Apple App Store now, or check out my Kickfolio page for screenshots of the app.

UPDATE: v1.1 will be available shortly with a fix that increments the syllable count as well as the stutters count when the 'stutter' button is clicked.


Wednesday, 12 September 2012

Smooth Talker finally submitted to the Apple App store

After quite some time developing the app - Smooth Talker - our stuttering app, has finally been submitted to the Apple App Store!

The app allows Speech Pathologists & Therapists, as well as parents of those that stutter - to measure their syllables to stutters ratio.

Smooth Talker has been submitted to the Apple App store
Smooth Talker - submitted to the app store!



Wednesday, 5 September 2012

Pair programming rocks socks (and you should try it too)!

Since starting on our idea for Contactable a few days ago, Herb and I agreed that we would pair program using XCode and Objective C.

This is my first real foray into pair programming - and it's awesome! Ideally, we would have two keyboards & mice of the same type, but we're currently shuffling with one set (and its particularly mind warping for Herb, since I drive the mouse with my left hand!). Even then, the productivity has been amazing - I don't think we could do this 100% of the time though, so its great that we take a break in the afternoon and separate to work on different things.

Herb, being a far more accomplished programmer than I am, probably gets frustrated with my fumbling, but you can teach an old dog new tricks and I'm looking forward to getting back to coding over the next few weeks.

After a quick search online, and past experience at Tyro, it seems the generally accepted rule is to have 3/4th of your time spent in a pair, with the other 1/4th devoted to individual learning & administrative tasks.

I don't know how feasible that will be in the long term, but it certain seems like the best option to rocket productivity for a startup - so if you aren't already - give pair programming a go!

Onwards and upwards.

Monday, 3 September 2012

Unconvention Sydney September 2012 - reflections

This Saturday's #uncon was an amazing event - over a thousand people turned up for a day's worth of talks on entrepreneurship, building better businesses and a pitching competition.

Hosted by "The Entourage" - Andrew Morello & Jack Delosa - the event was immensely useful for a budding start up bum like myself. I wouldn't hesitate to say that other events would charge hundreds of dollars for an event that provides much less value - and thanks to the sponsors, this event was entirely free.

Here's a brief breakdown of what stood out on the day from each speaker for me (my notes were sparse, so the list is no indication of how well a speaker actually presented).

Andrew Morello
  • Do not be afraid to try something that makes you uncomfortable.
  • observation: Excellent presentation skills are gained with practise (Andrew was a fantastic MC).
  • observation: Ensure that you think about your customers from their perspective.
Sarah Riegelhuth (@SarahRiegelhuth)
  • Use what's unique about you to differentiate yourself, your product and team.
  • Outsource book keeping to sites such as xero.com.
  • Manage PR through sites such as Source Bottle.
  • As a business owner, you are wholly responsible for maintaining leadership and espousing your corporate values. 
Jack Delosa (@jackdelosa)
  • Implement systems such that YOU do not restrict the growth of your own company.
  • Strategic partnerships are key, things to think about:
    • where do my customers go before they buy my products (e.g. McDonalds before the movies)?
    • where do they go after buying my product?
    • who services the same market but has a different service or product?
  • Raise cash only to:
    • Accelerate demand and,
    • Facilitate existing growth.
  • Be wary that growth devours cash.
  • When investors look at your start up, they look at:
    • the inherit risk in the business,
    • the experience of the entrepreneur and their advisory team,
    • a proof of concept or prototypes,
    • the speed at which they will recoup their investment,
    • how the business will make money.
  • Business is a team sport.
Peter Davidson
  • observation: Being in California helps, since most deals take place due to serendipity.
  • Focus on changing the world, but be realistic about the products you are going to build.
  • Raise more than you ever need.
  • VCs are humans - don't put them on a pedestal & not all of them know what they are doing.
  • Get to know your VC before committing.
  • A great team in a crap market > great market, crap team.
  • Dont expect VC's to understand or validate your idea. You are the domain expert, convince them to come on the ride.
Matt Rockman
  • Ensure you have a customer focussed culture (I liked his example of seek.com's joint Christmas parties with customers).
  • Do what you can to maintain your own unique corporate culture.
  • Be paranoid & always keep an eye on the competition.
  • Be bold with marketing (citing his use of controversial figure Alan Bond).
  • Be courageous in how you manage your team.
  • Be patient, because earning revenue takes time.
I'm sure there's a lot that's missing here, but if I remember over time, I'll definitely add to this list. 

Awesome conference all around!

Dave Cupples over at FatCowBusiness has a far more detailed summary of the #uncon that he's posted online, definitely check those out - they're an excellent summary of the day: