The SoCal Tech Group was put together roughly two years ago by me and a few of my buddies from PhatNoise. Its mission is to help connect technical folks in the Southern California area with each other and the venture community. I am a big believer in entrepreneurship and in LA.

I remember from PhatNoise’s early days just how many times I was traipsing to the Bay Area with pitch in hand, only returning with serious doubts as to why exactly why the company remained in LA. It was NOT an asset to be in LA: all the good developers move to SF (so I was told), and being an LA startup gave most folks in Nor Cal the vibe of a slightly sleazy Hollywood party. On many, many occasions did I consider returning to the Bay Area — it’s where I grew up, after all.

In the end, for PhatNoise, being in LA did have its advantages; we had both automotive companies and the port of Long Beach nearby, which is a great asset in the hardware business.

But I also knew why I personally wanted to continue to be in LA. First, it’s a dynamic and massive city. It is a mashup of cultures and industries, all lying on top of each other. It’s also a very creative city – the arts scene here is incredible. This is a place where, on a daily basis, you’ll likely be healthily confronted by many opposing opinions, something unfortunately missing in most Bay Area circles.

Second, the academic pedigree of Southern California seems to be under appreciated. While certain communities in the US are synonymous with higher education, SoCal can’t shake its sunny reputation. But the fact is LA has been home to some of the most influential people in technology. You can start with the legendary physicist Richard Feynman at Caltech, or maybe the more recent computer science greats such as Leonard Kleinrock and Judea Pearl (both from UCLA). The founders of both Blizzard entertainment and Broadcom were UCLA students, and Andrew Viterbi – the father of modern cellular communications – went to USC. Goto.com, from Pasadena, was the first to develop a search advertising business model. The communications systems responsible for letting the Mars rovers talk to earth were designed by a good friend and classmate of mine.

But the physical scale of LA is a challenge. You don’t have the serendipitous run-ins that frequent many SF or Palo Alto startups. You can’t get an impromptu gathering of people at a coffee shop quite as easily as in other locales. So the community, in my opinion, remains fragmented and not interwoven with the city they live in.

So this posed a challenge to STG as well – how can the group break this habit of forming disconnected communities in LA? The answer would lie in social media.

When we started, socaltechgroup.com was largely brochure-ware. A set of static pages with little call to action, and since all of us founders has such busy lives, we were not able to devote much time to make the site more effective. Well, starting in the fall of 2008, I started from scratch with Wordpress as a foundation. I made pages for each of the consulting members. I made articles about the group and any events we were doing. I created Facebook and Linked In groups. I installed all the standard analytic and SEO tools for Wordpress. In short, I made ourselves known – and as a result, we have tripled our membership in just four months. We’ve been able to make great relationships, and we’ve started to help out the community by helping market events like Startonomics.

This is where I see a very practical future of the web: maybe just like in Blade Runner, Los Angeles is a look towards the city of tomorrow. A place where we live in such close quarters, yet we don’t interact. We crash, but we don’t connect. The web may be the connective tissue of people, serving groups as niche as startup entrepreneurs or as broad as entire societies. It is less about mimicking the offline world, and more about bringing people and ideas together. Regardless, this is just the beginning, and I am proud to be part of an effort that is working on this right now, and right in my backyard.

While I was PhatNoise I witnessed firsthand the introduction of Apple’s iPod.  This was really the first product that directly competed with ours, yet for years investors had always warned us of companies like Sony or Philips just running us over.  We always contended that while Sony had 100000x the money and resources of our little company, it didn’t really matter — they weren’t wired to make a compelling portable music player.  As my business partner likes to say, they wouldn’t even know the right questions to ask.

Of course, the MP3 was a completely disruptive technology, and the company best positioned to introduce it to the masses was Apple.  So when I came across the Foundry Group’s post about Digital Life it got me thinking about this problem in the realm of the living room.  They correctly note the challenges in building a product that makes enjoying digital media in the living room an intuitive process for everyone: hardware is not for the faint of heart, and must be dead simple to use right out of the box.  And as mentioned, giants such as Apple, Sony, and Microsoft have all doused the problem with hundreds of millions of dollars with no real great strides in product innovation.  So, what’s the problem?

Just today there have been a bunch of posts about boxee raising $4M.  From the videos, Boxee looks quite slick, and I know it is built on XBMC which has been in the works for a while.  I haven’t played with it, but I have played with (in order from newest to oldest):

- AppleTV
- Meedio
- Windows Media Center
- MythTV
- some other crappy ones I can’t remember

Hell, even PhatNoise made one* for browsing audio content on a TV.  Most of these are varying attempts at getting files (either local or remote) to stream on your TV.  A lot of attention has been placed on things like browsing big lists of files, fetching metadata, and making sure the software can put on the screen whatever could be put there (photos? sure!  the weather? why not!).  But let’s say you made the perfect software/device thing, and it synced and played beautifully.  The AppleTV has lots of flaws, to be sure, but it is pretty good at just this task.  So why is it kinda ho hum?  Steve Jobs’ hobby, dare we say?

One of the things I realized when I first held a Compaq reference design of a harddrive music player back in 1998 was that it would change everything.  It was totally a new mode of operation, to have that much music wherever you were.  There was no other practical way of achieving the same result. It was just a matter of the right company to educate the consumer and it would take off.  But I can’t really find a parallel with so many of these “digital living” products.  While it is kinda cool to have a large movie collection sitting there on the screen, the same thing is achieved with a DVD player and a bunch of discs.  We’re not talking about portable here, so the fact that DVDs are clumsier doesn’t make much of a difference.  How about streaming a movie from the web?  Well, most folks have some sort of Pay-per-view or On-demand through their broadcast provider.  And while not cool, their QOS usually bests what you could get online.

Lastly, what about stuff that is web only, like YouTube?  Here we are getting closer, since there is no “sneaker-net” way of getting that stuff on your TV.  But what does it mean to watch it on your TV (beyond the obvious, smarty pants)?  What is the user’s behavior, their preferred use case?

Well, this post from Brightcove CEO Jeremy Allaire (and the associated TechCrunch coverage) I think  describes precisely the problems to date, and outlines what a use case would look like.  However, I strongly disagree with the approach: getting standards bodies to define such a set of requirements.  This will go nowhere really quickly, guaranteed.  Why?  The people at the table, CE companies, Web companies, PC suppliers, infrastructure companies, etc. do not have the DNA to devise such a product.  It just isn’t in them.  It would be about as disruptive as Sony’s MP3 players or Microsoft’s Media Center.

We are looking at a gap here.  The really awesome, game changing thing lies in between what all these companies do well.  No standards body is going to magically solve that.  So what will it be?  Maybe Boxee is on to somthing, or maybe Apple will make an SDK available for AppleTV like they did for the iPhone?  What I do know is that standards bodies may be largely ineffectual in the future — too slow and too many competing agendas.  This is a winner take all game, folks.

(* I can’t believe that manual is still there.  I had a hand in this one, probably the only product manual written entriely in LaTex!)

A basic tenant of configurable computing, and the one that got me hooked as a senior in undergraduate engineering, as the idea that the lines between hardware and software could be blurred. Software had always been tailored for the hardware that it ran on, whether it was done that way by hand or if left up to the compiler. Well, what if you could design hardware that better fit the needs of a particular application, automatically? My idea was to use the well-known tricks of scheduling for VLIW machines and use those as a set of “hints” for the what the ideal hardware could look like.

The class professor wasn’t too hot on this idea, but let me do it anyway ‘cuz I’m tenacious. So for the next few weeks I poked around an open-source compiler and messed with making a mythical hardware design out of the compiler’s scheduler and ended up with some interesting results.

For example, most DSP’s have a single instruction multiply-and-add. This is because the most common thing they do is add and multiply (you can find a bunch of these in the my previous paper). Well it turns out that you get only modest savings from doing that. My research showed that for most implementations, making faster individual multiply and add units performed better than making a monolithic one.

The results were interesting enought that the prof convinced me to submit my work to a handful of conferences. Ha! This was literally a stretched out homework assignment and he expected it to be accepted at a research conference? Well, it did, and to one of the most prestigious ones — ICCAD. Here’s a link to my paper, the abstract and embedded copy can be found below.

Abstract

Many previous attempts at ASIP synthesis have employed template matching techniques to target function units to application code, or directly design new units to extract maximum performance. This paper presents an entirely new approach to specializing hardware for application specific needs. In our framework of a parametrized VLIW processor, we use a post-modulo scheduling analysis to reduce the allocated hardware resources while increasing the code’s performance. Initial results indicate significant savings in area, as well as optimizations to increase FIR filter code performance 200% to 300%.

There is a lot I want to write about PhatNoise, since there is a lot that happened.  Literally a garage startup, PhatNoise ended up running for over six years, with half a dozen automotive deals, and finally a successful sale to Harman International.  Whatever I write, however, I would like these notes to be considered both a post mortem and a straight historical accounting of what went down.

PhatNoise began as a question I asked myself: could I put a little computer full of mp3s in my car by replacing (and mimicking) a regular old CD-Changer?  Only one way to find out – reverse engineer the communication path in the car, build the hardware to interface it to a PC, and off we go!
Our home base was a crappy apartment in west LA, which happened to be just up the street from the infamous DEN.  Here’s Dan Lau, my roommate at the time, where it all started:

What we had hobbled together wasn’t much, and it would frequently break down in a plume of smoke, but it was enough to convince us that having thousands of songs in your car was damn cool.

Keep in mind this is late 1998.  It was about six months before the original Napster was released, and about three years before the first iPod.  I would often joke that the first half of any meeting we would have with an investor or hardware vendor was devoted to explaining what mp3s were and why anyone would care.  But this demo was enough for us to realize the potential, and we thus set out to build a dedicated solution to replace the jungle of wires pictured above.  My buddy Dan and I finished up our Master’s degrees at UCLA, and founded the company in July of 1999.

Pizza and Ikea

The company was largely conducted out of our living room.  Most of our hires were friends of ours at school, many of whom had not graduated yet.  The operation was run on Ikea furniture (the first job of any new hire was to build the desk they would call their home), and lots of pizza dinners.  Folks would head over to our pad (the “PhatPad”, the first in a long line of Phat* names), and bang out various aspects of the concept.  There wasn’t much of a plan, nor did anyone have any relevant experience beyond being smart engineers.

Creating a small Linux computer to interface with cars was the main idea, but a bunch of other problems needed to be solved along the way.  The biggest was moving music from the PC to the car.  Our original concept was for computer to house a hard disk drive full of music, and you would shuttle the whole computer to and from the car.  At the time, though, USB-bulk storage devices were *just* coming out on the market.  Using painfully slow first generation USB, these were designed to accommodate external hard drive enclosures.  Our idea as a bit more ambitious – house just the drive in a small cartridge that could easily slide into bays or docks.  Here’s what very early sketches looked like:

The computer itself (the “PhatBox”) was originally thought to look not much different from CD-changers, such as this sketch below:

This idea was ditched when our contract manufacturer convinced us that using a single piece of extruded aluminum was cheap to tool, and looked way cooler.  With our concept firmly in hand, we were able to raise enough venture money to actually pay people and to move the operation out of our apartment.

Prototypes

Speaking of which, this is probably the biggest lesson in building a new hardware product: get really comfy with the contract manufacturer.  You gotta choose them as if you were going on a long, dangerous, and ill-planned camping trip together.  This partner will need to take on a lot of risk if you are going to do this on the cheap, and yet they will need to be capable enough of really running the show.  Fortunately, we did have a great partner that believed in our vision, and part of what this partner did was create manufacturable 3D models of our sketches and quick-turn SLA prototypes.  Here they are:

Pretty damn close, wouldn’t you say?  Here’s how the cartridge fit in the cradle, with a giant CRT monitor for scale:

The cartridge never did quite perfectly fit in the dock, considering there was a 40-pin connector there and all.  Rigorous mechanical engineering is one of those things that you just gotta pay a lot for.  If this looks familiar, well let’s just say we inspired a few folks:

Iomega, really, we’re flattered!

Enough about plastic, though, onto electronics!  Our idea was always a small computer, capable of playing back thousands of mp3s, interfacing to a variety of automotive networks, and easily software upgradable.  In order to fit our cost and mechanical constraints though (more on that later), it meant finding a processor “just good enough”.  It also meant pretty much designing and building it from scratch.

The just good enough processor turned out to be an ARM7 running at a whopping 74MHz from Cirrus Logic.  And just as important as being cozy with your manufacturer is to be really chummy with your major chip vendors.  Processors are complicated beasts — even little ones like this — and you’ll need all the help you can get.  Cirrus was always a great vendor for us.

Software was done all in house: a stripped down Linux kernel, software audio decoders, and daemons to manage everything.  A smart decision we made was to include a second, smaller microcontroller on the board to handle interfacing with the vehicle, power management, and in-system updates.  It acted like a bridge between the PCB and the Linux OS.

I want to make the point that this was all very fancy stuff for a music player.  Every other device on the market used a dedicated IC for decoding MP3s and doing everything else on the device.  We were the first to employ a full OS and software decoders in such a product, and we got press simply for that fact.

The boards took quite a long time to get working properly, but this basic design would last us for years.  Here’s a shot of an early PCB:

Along the way, the company crammed as much as possible into the space and time we had.  That’s me on the right – hey at least we had a view!

Although one downside is that at 4am it gets quite cold in an office building with no heat on, even in Los Angeles:

Notice all the crap in the back – oy!

We wouldn’t have had much of a music system if there wasn’t a tool to effectively rip CDs and manage all that stuff.  As if there wasn’t enough to do, we made that as well.

It was not the first music management tool, but one of the only built around a database to manage thousands of tracks, and with a dedicated interface to sync it to the portable drive.  And since all the cool apps had funky skins, ours did too.  One of our early users called it “garish”, and the name stuck!

The Big Show

After about 15 months from inception, we were ready for our first big show: the MP3 Summit, sponsored by the then huge mp3.com.  Most of the companies were either software services (luminaries such as Scour and moodlogic), or large hardware companies such as Philips and Thomson.  And then us.  We had a barely working system, held together with tape and with a soldering iron always nearby.  One of us was permanently on duty to reset crashed demos.  But none of it mattered, since the crowd was in awe:

We were easily the most popular booth at the show, because people saw something for the first time in the flesh: thousands of songs, accessible by the stereo they may already have in their car; a portable harddrive with USB connectivity, and a database driven music management software for the PC.

Was it worth it?  Well, we won best of show.  The company went on to strike deals with most major automotive suppliers, and to develop several variants of the system, including a home version and a multimedia version for General Motors.  We have PhatNoise alums at both Apple and Google.  Our main competitor (and major source of inspiration) went on to become the hardware chief of the Apple iPhone.  Here’s Hugo and myself at that summit:

About six months later, when rumors were swirling around a totally new product line from Apple (a week before the iPod was announced), a Wired reporter asked me for a quote.  I didn’t think Apple was doing anything directed at cars, and to this day most interfaces for iPods are pretty lame.  But Apple did take what they knew was a game-changing technology (MP3s), and spent the money to cross the chasm to the masses (more on that another day!).  I’d like to say that our company did some trailblazing as well, and was an experience I wouldn’t have gained any other way.

So you wanna get into the electronics business?

As I mentioned earlier, there is a lot more to write about PhatNoise, but since you don’t run into too many consumer electronics startups, I thought I would spend some time on some humble advice.

Try to avoid hardware. Really, it is awfully expensive and time consuming.  But if you got this far you are probably hell bent on making hardware an integral part of something new, so…

Use the biggest building blocks possible. It’s like Duplo versus Lego: you maybe able to make something more intricate with Legos, but that’s reserved for the bigger kids with bigger wallets.  For the Techcrunch tablet project, for example, they should really just book a weeklong trip to south China, look at lots of existing models that come close to their vision, and weigh the pros/cons of modifying it.

Weigh the opposing forces. There is a saying: fast, good, or cheap – pick two.  Well, I can make this a bit more precise for consumer electronics.  The NRE cost, final price, performance, flexibility, and mechanical constraints are five dimensions that generally oppose each other.  For example, tight mechanical constraints will make the product NRE cost more.  But lowering the NRE cost will likely raise the final price.  Increasing performance (faster processor, longer life battery) stresses the mechanical design (faster processors need better heat dissipation, longer life batteries are usually just larger ones).

Find an original design manufacturer (ODM) that can do it all, and let them do it all. Let’s say you violate this rule and hire an amazing mechanical engineer outside of the ODM to design an awesome dock for your web tablet.  The tablet is largely the ODM’s design, and the dock is from this super fellow.  And when you have the first prototype and it doesn’t dock correctly one out of a hundred times, who is going to find out what’s wrong and make the appropriate change?  Both the mechanical engineer and the ODM will point fingers as at you.  Let the ODM do all the work and manage their own subcontractors.

Make sure the hardware-software connection is clear. Lots of folks feel they have a stronger handle on the software than on the hardware, and thus make that the dividing line.  However, the hardware and software need to work in concert.  If your web tablet is supposed to have a “sleep/resume” feature, what is the mechanism of waking it up again?  Do all the peripherals get reinitialized correctly?  Are the things that are supposed to be non-volatile really safe?  How about updating the software?  Can you update the operating system?  If so, what happens if that update fails (and it will)?  If you are not careful you will be left with a cool paperweight.

Where is the product going to be sold, and how is it going to be used? This is painfully tricky – FCC is required in the US, CE in Europe, and it kinda varies everywhere else.  Do you know what the packaging requirements are like?  Do you have French everywhere in order to be sold in Canada?  Who is writing the user’s manual?  What are the environments this is to be used in?  What are the requirements for temperature, humidity, mechanical shock, EM radiation, EM immunity – and who’s going to test all this?  If you think this should all be “ok” the next time you try out that LCD panel that’s really cheap, think again — these are the exactly the areas the vendor is likely cutting corners without you noticing.  Lastly, if the product doesn’t pass FCC/CE testing, who knows how to resolve it?  If you have any “custom” electronics made, the ODM will just hand the problem right back to you.

I’ll have much more to write about both PhatNoise, building startups, and other aspects of the consumer electronics business.  This list above is really just the tip of the iceberg, but this post is already waay too long.  In the meantime, please don’t hesitate to contact me through the comments below or via the methods on my about page.

I remember literally staring at the math behind my lab’s latest research work.  I was a student at UCLA, and our lab just was on a high from having made the cover story on Scientific American.  The lab worked on the (then hot hot hot) topic of reconfigurable computing.  Specifically, we worked on machines that would perform image correlation very quickly; my specific project performed at 12FPS running at just a 16MHz clock!

What struck me was that what the machine was doing could represent a more generic way of how machines performed mathematics — vector math, to be precise.  So over the course of about a summer, I wrote the following paper (I also wrote the actual C++ code that did the work, but can’t track it down!).  Here’s what that code could do:

Re-write the following equation, using simple arithmetic, so that all multiplies are done using powers of two, and with the fewest number of adds or subtracts:

There are 7 plus signs already — how many more will you need so that there are only powers of two? (Answer at the end of this post!)

This is exactly the sort of optimization that is needed to minimize hardware design complexity (a multiply by a power of two is just a shift, after all).  I loved this project because it transformed this algebraic problem into a graph coloring problem, and by doing so, allows computers to do all the dirty work.  For example, if you gave the code a fixed point representation of the DCT, it would optimize and transform the 8×8 matrix multiply to something that approximates the widely used version by Feig & Winograd.  Except it took the computer just about a day to do it.

Abstract

This  paper  presents  a method,  called  multiple  constant  multiplier trees  (MCMTs),   for  producing   optimized   reconfigurable   hardware implementations  of vector products.  An algorithm for generating  MCMTs has been developed and implemented, which is based on a novel representation  of common subexpressions  in constant  data patterns.  Our optimization  framework covers a wider solution space than previous approaches; it also supports exploitation  of full and partial  run-time  reconfiguration  as well as technology-specific constraints,  such as fanout limits and routing.  We demonstrate that while distributed arithmetic techniques require storage size exponential  in the number  of coefficients,  the resource utilization of MCMTs usually grows linearly with problem size.  MCMTs have been implemented in Xilinx 4000 and Virtex  FPGAs,  and their  size and speed  efficiency  are confirmed in comparisons  with  Xilinx  LogiCore  and ASIC implementations  of FIR filter designs.  Preliminary results show that the size of MCMT circuits is less than half of that of comparable distributed arithmetic cores.

Download the full paper (PDF), or browse through the paper below (it looks blurry cuz the font wasn’t properly embedded, crap.)

If you are still reading you must be dying to know the answer — well, in the paper I describe how to use only 10 additions and only powers of 2 multiplies. See if you can do it by hand!

I have long admired how folks working in artistic or design fields have been able to neatly wrap up their work in largely a visual format — whether it be old school vellum and paper, or the latest 3D visualization tools.  After many years hanging with my sister in architecture studios and witnessing this firsthand, I have thought about doing the same with my own stuff.

Well, turns out it is not so easy…if you are an engineer / software developer / even product designer, it seems like your work output is not easily archived; screw the work flow, even the process of working is hard to archive.  For example, if you write software, one could argue that archiving it is simply a matter of source control.  You would have all the changes to the source, and build scripts to try out the final thing.  But what if the code doesn’t really work — like it crashes before it gives any sort of indication that it did something?  And what about the thought process that got you there, is that captured in your source?  Also, who is the intended audience?  What if there are really interesting ideas in your project, but your audience doesn’t understand the language it is written in?  I guess the question is really how much more work do I have to do for all this to be reasonably archived by me for others?

Well, I thought I would tackle an easier task — archiving technical papers I have written.  It’s a paper, what’s hard about that?  Well, like many technical papers, I wrote mine using the fantastic LaTex markup language.  Really one of the first widely used markup languages, LaTex is extremely powerful, produces beautiful photo-ready output, and dutifully follows the “separate the content from the design” ethos.  Any because it is a markup language, the LaTex community has made a number of LaTex output converters: HTML, PDF, etc.  However, it really hasn’t been maintained — this HTML converter has a last revision of 2001.  How about LaTex to Wordpress?  Not a chance. *

But more to the point, why do I think publishing my work via this blog is the right way to archive it?  Well, blogs get searched, and blogs get linked to, and blogs get syndicated to you, the reader.  This means your new document storage and retrieval system is a web search.  And if my blog page goes down for good?  Well the contents of this site are frequently archived duplicated by crawlers.  This may be an obvious point to some, but if I don’t do this, my published work will fall victim to the one-way black hole of “academic portals”.

Go ahead, search for one of my papers using google scholar.  Just try to find a full copy of it without opening your wallet.  That’s right, IEEE, ACM and most others charge to gain access past the abstract.  I have nothing against these associations helping offset costs, but putting a fortress around the information seems to work against the whole academic process, right?

So on this site I’ll post abstracts, links to full PDFs (that I host), as well as upload the contents to services like scribd.  I’ll try to give an introduction for each, to give the content a broader audience, so please give them a gander.  But regardless — copy away, Intertubes, copy away…

* I would like to give props to this great LaTex-Wordpress resource page, which allowed me to put at least a few equations (but not entire document conversions) quite easily inside posts.

Image from this fellow’s impressive record archive

He’s already made me a very proud dad…here’s to you kid!

It’s ICQ all over again

Making a communication service profitable is not for the faint of heart – direct response advertising in such a social medium performs poorly, and charging for the service generally only works when you can be cheaper than the alternatives. Most importantly, a communication service won’t have a chance surviving unless there are just gobs of users. So why a new service comes along and tries to build a closed system, preventing it from attracting users in every way possible, is beyond me.

Twitter is the ICQ of this decade, folks: a great communications idea that doesn’t make money and doesn’t work very often. And both Twitter and ICQ have chosen to be closed. When ICQ gained enough attention, it begat competitors like Yahoo! and MSN to make their own (closed) IM systems. “Hey I’m on AIM! Oh noes, I’m on MSN! Guess I have to make an AIM account and run both!” Retarded, yes? So how is this different from “follow me on twitter – no wait screw twitter, follow me on plurk!”

So I want to expand my previous plea of making parts of twitter open source: not only is it the best way to build a scalable service, but it allows twitter and its competitors to focus on how well it designs a service for its users. There is nothing sad or unfair to twitter about this; they are toast if they don’t do it.

The Conversation has started

First off, go read Eran Hammer-Lahav’s great set of posts on the topic. I agree with both Eran and Scott Hanselman that the computational challenges may be alleviated by focusing on smarter clients and a more sophisticated API. If I blog and produce a piece of XML, and your reader pulls that XML on demand, then the information is simply duplicated. Is there a role for a centralized server? Well, you could build an email gateway (you email your blog, and it can email you updates of what you follow), an SMS gateway (that you may have to charge for), gateways to other protocols, and of course you can build a web-store of all this data. Maybe these are the things Twitter Inc. should focus on (still not sure it’s a business though).

While we are separating the science from the service, let’s ensure the marketplace is interoperable. So why can’t I follow bob@twitter while posting to dan@pownce? Maybe a URL is all that is needed to identify the user from the service.

Sometimes you have to open up to your hard problems

Lots of people are talking about this — I suggest you follow how the conversations progress.  My curiosity lies in what this has to do with open source and business.

Two proven capabilities of making any project open source is (1) if it is a worthwhile endeavor, you get really smart peer reviewers to help solve hard problems, and (2) it takes care of the not-profitable-but-necessary things like interoperability so that business can focus on how they bring value to their users. Open source has let products ranging from TiVos to iPhones focus on what consumers care about (which isn’t file systems and schedulers). Competing on uptime is no way to attract a mass audience, so why not let the fundamentals be a known problem and allow businesses to focus on building unique value? Let’s separate the service from the science.  As popular as Twitter is, it still is far from mainstream.  Twitter needs to cross the chasm, get people to understand the service, use it in as many ways as possible, build/support all the periphery services to it, sign carrier promo deals if they need to.  All of this is really hard work, and it is assumes the underlying mechanisms are in place and ready for the scale.  Once you have Skype-like numbers, then you can do some interesting brand advertising deals (”follow the new chevy on twitter!”).

A call to action

So this post is not much more than a call to action. I don’t know how to build the science and I am not interested in starting such a service — that’s up to all of you. But good business is always about learning from past mistakes, so let’s not let 10 years in Internet time obscure those lessons.

P.S.: I gotta plug Twoorl: Open source, nicely scalable, and written in a single day by an Erlang stud.

(Cat nap courtesy of a napping twitter image)

]]> http://dan.benyamin.org/2008/06/12/open-source-twitter-and-you/feed/ 0 http://dan.benyamin.org/2008/05/21/why-isnt-there-an-open-source-twitter/ http://dan.benyamin.org/2008/05/21/why-isnt-there-an-open-source-twitter/#comments Wed, 21 May 2008 19:24:47 +0000 Dan http://dan.benyamin.org/?p=18 I am not a twitter user, as my stimulation / activity ratio is already waaay too high the way things stand, but I sure do get a lot of secondhand twitter smoke. Just wading through techmeme and techcrunch to find stories not about twitter being down is getting tiresome…

So why hasn’t someone developed an open source Twitter? It seems like a natural direction — twitter is a communications system, and a fairly generic one at that. A great idea (not sure if it is a great business), and one of the best ways its users can ensure its reliability and uptime is to reverse engineer it and license it under open terms. This would not only allow for a distributed system that would naturally scale, but would allow the protocol to be picked upon and augmented/improved as need be. Really a similar history to Gnutella – if Gnutella was kept under wraps there would be no way that it would power the majority of all P2P traffic on the web.

This is a short post and I’d love to do more research, but thought I would throw the question out. I won’t pretend to know the technical details of twitter, but I honestly can’t see any other viable outcome…

(Image courtesy of luc legay)

Gladwell and his contemporaries argue that while Duncan’s results are interesting, they stand by their “real-world” results.  Duncan, in-turn, states that their results don’t necessarily point to a root cause, with conclusions largely theoretical.  Tempers flare.

As I am embarking on a real world modeling effort myself, it struck me as odd that such contentions might really exist: both parties are right, and what the article really is pointing out is a very common dilemma in any engineering endeavor.  Models are, by definition, inaccurate.

Models are used as a utility, and well-designed ones make approximations and assumptions that are understood and accepted.  I’ll use an example from my old acoustics days.  If you wanted to build a sealed wooden box with a speaker stuck in it, you can use the following formula:

This is not as bad as it looks, as is the volume of the box, and the other values come with the handy spec sheet of the woofer you buy.  A few simple punches in the calculator and some wood glue, and you have a decent sounding speaker!

This is a great model, because it boils down very complicated acoustics into high school algebra.  If you studied acoustics you will find more elaborate models at work, but even those are inaccurate.  You can involve thermodynamics and even human hearing physiology, but there are diminishing returns for all this work.  The equation above gets you 90% of the way…reams of additional study will get you maybe a few points more.  That equation bakes in literally decades of work, with its principle authors essentially saying “trust us, it will sound fine.”

So, in the case of studying the viral trends, Duncan’s models probably serve his needs of predicting the outcome of Yahoo’s advertising programs, within a certain ballpark at least; but they are not perfect.  Gladwell and others, however, are also correct in basing hypotheses on the observed outcome of various real world studies; but they cannot prove it.  There usually exists a gap between observed outcome and predicted outcome in natural systems.  I say “usually” as there are exceptions, and these exceptions have largely changed our world — something I’ll have to write more about later!

(Photo from the source article.  The pretty printed math courtesy of the great wpmathpub plugin!)

]]> http://dan.benyamin.org/2008/05/14/modeling-the-natural-world/feed/ 0