In the table above, note the drastic growth of input data analyzed and output data generated. The number of actual MapReduce jobs has also grown significantly and reached 10,000 in September 2007.

Dean also presented an interesting graph about the frequency of commits of new MapReduce jobs into the repository - as you can see there are months when the number of new projects goes through the roof, followed by a spike.

The reason? Summer interns.

Complete set of slides is available from Yahoo! Research, which organized the Data-Intensive Computing Symposium.

1. A drop of 100ms in page rendering time leads to 10% in sales on Amazon. A drop of 500 ms leads to 20% less traffic to Google.
2. Make fewer HTTP requests - combine CSS and JS files into single downloads. Minify both JS and CSS.
3. Combine images into CSS sprites.
4. Bring static content closer to the users. That usually means CDNs like Akamai or Limelight, but sometimes a co-location facility or data center in a foreign country is the only option.
5. Static content should have Expires: headers way into the future, so that they’re never re-requested.
6. Dynamic content should have Cache Control: header.
7. Offer content gzip’ed.
8. Stoyan claims nothing will be rendered in the browser till the last piece of CSS has been served, and therefore it’s critical to send CSS as early in the process as possible. I happen to have a document with CSS declared at the very end, and disagree with this statement - at least the content seems to render OK without CSS, and then self-corrects when CSS finally loads.
9. Move the scripts all the way to the bottom to avoid the download block - Stoyan’s example shows placing the javascript includes right before </body> and </html>, although it’s possible to place them even further down (well, you’d break XHTML purity, I suppose, if you declare your documents to be XHTML).
10. Avoid CSS expressions.
11. Consider placing the minified CSS and JS files on separate servers to fight browser’s default pipelining settings - not everybody has FasterFox or tweaked pipeline settings.
12. For super-popular pages consider inlining JS for fewer HTTP requests.
13. Even though placing content on external servers with different domains will help you with HTTP pipelining, don’t go crazy with various domains - they all require DNS lookups.
14. Every 301 redirect is a wasted HTTP request.
15. For busy backend servers consider PHP’s flush().
16. Use GET over POST any time you have a choice.
17. Analyze your cookies - large number of them could substantially increase the number of TCP packets.
18. For faster JavaScript and DOM parsing, reduce the number of DOM elements.
19. document.getElementByTagName(’*').length will give you the number of total elements. Look at those abusive <div>s.
20. Any missing JS file is a significant performance penalty - the browser will browse the 404 page you generate, trying to see if it has valid <script>s.
21. Optimize your PNGs - check out pngcrush, pngoptimizer
22. Optimize JPEGs - jpegtran
23. Make sure you have favicon.ico - generating those 404s will be expensive, plus once you have it, it’s cache-able.
24. Toolkits for measuring page loads: AOL PageTest, FiddlerTool HTTP debugging proxy, IBM Page Detailer instrumentation tool, YSlow, and Firebug are suggested in the presentation. My personal addition to the list is Charles that has been recommended by a colleague.

And here’s the whole presentation, although it’s not possible to follow links from Slideshare slides.

• a high precision screwdriver for hard to reach bolts
• a replacement analog stick if yours is worn out
• button replacement kit, if yours are dirty or unusable. If you only need specific buttons replaced, check out this kit.

Unscrew three bolts on the back of PSP to separate the front and the back of the device. The last screw might be tough to find, and it is under the battery, which you need to take out. There’s a protective seal that warns you about warranty being void if you remove it. If you have any kind of warranty left on the device, you should probably have it replaced instead of cleaning it out yourself.

Separate the front panel of PSP, and remove it. It contains many small buttons such as Start and Select, which could fall out, and get in the way. The front panel frame hardly needs any cleaning, so it’s safe to just put it away for the time being.

The back of the panel, however, contains the analog stick, and 4-directional button. The analog stick can be replaced, if you have a replacement handy, or just cleaned out, if it’s just the matter of dust and a few odds and ends getting in. If you remove it, that’s the only electronic part on the front panel. The rest of it can have a date with Mr. Windex for brighter shine.

Replace the analog stick of 4-directional button, and you’re pretty much done. Check out the top buttons, R and L, that are not used in all the games, and therefore might be in different state of wear and tear. Those can be washed and cleaned out as well, nothing but white plastic there.

Time to put the PSP back together, and remember - no spare parts.

For a pretty small budget, and 20-30 minutes of work you have a good-looking shiny gadget back in shape.

You begin by defining your test cases in an XML file. Test cases are where you specify the requests (url, method, body/payload, etc) and verifications. Server responses can be verified by matching content to regular expressions and with HTTP status codes. You can adjust the load settings in the workload controls on the GUI before you start a test run (number of agents, request intervals, rampup time, test duration). These settings enable you to model tests based on various load scenarios. At runtime, the cases are loaded and passed to the load generating engine. Agents are dispatched and run concurrently to send HTTP requests to your web service. Real-time stats and error reporting are displayed for monitoring the test as it executes.

It’s a Python script with command-line and graphical interfaces.

register_shutdown_function is also useful for command-line scripts with PHP. Pretty frequently your script has to do some task like parse a large XML file, and the test examples when it was originally written did not account for the XML file possible being huge. Therefore your script dies with like 23% completion, and you’re left with 23% of the XML file parsed. Not ideal, but a quick duct-tape-style fix, would be to introduce a register_shutdown_function call to system(), to which you pass the script itself.

If you happen to keep track of which line you’re on while parsing, you can pass the line number as the first parameter to your own script, and make it start off after that 23% mark, or wherever it died. The script then needs to be launched with 0 passed as the first parameter. It will run out of memory, die, launch register_shutdown_function, which will launch another copy of the script (while successfully shutting down the original process) with a new line number, which will repeat the process.

Again, this is a duct tape approach to PHP memory consumption issues while working with large data sets.

The push-up is the ultimate barometer of fitness. It tests the whole body, engaging muscle groups in the arms, chest, abdomen, hips and legs. It requires the body to be taut like a plank with toes and palms on the floor. The act of lifting and lowering one’s entire weight is taxing even for the very fit.

Then there’s a blog post on NYT site as well, featuring 93-year-old Jack LaLanne, who still incorporates push-ups into his daily workout. They link to push-up calculator, which says that an average 27-year-old should be able to do 37 push-ups on average, and above 50 in a single session to be considered excellent. Google Video has a few videos on what’s considered a proper push-up.

How long would that query take?

Well, for one, a trip to the post office (20 minutes sound reasonable, but your mileage may vary). You have to physically request Form 6401, as there’s no option to pre-fill it online. So make it another 10 minutes at postal window.

Then it takes $5, as specified in the USPS money order rules. After that the filled 6401 travels to some place in Iowa, which would get back to you two weeks later in an official letter from some kind of USPS database query execution department.

Total cost: 2 weeks + 20 minutes + 10 minutes + $5.00

1. Their example lists the href enclosed in <link rel=”stylesheet” type=”text/css” href=”/css/c7y.css” id=”Main C7Y CSS”></link>. So is that a valid link? Anything in href qualifies as a link?
2. Does a JavaScript window.open qualifies as a link?
3. What about <a href=”http://www.yahoo.com” onclick=”window.location=http://www.google.com”>link</a> or any similar shenanigans? What qualifies as a link there?

Additionally, Emotiv has announced the commercial availability of its full SDK. The SDK has been upgraded significantly since it was first announced in March 2007 at last year’s GDC. The commercially available version of the kit now includes:

• 2 beta-version neuroheadsets
• Software toolkit that exposes the APIs
• Full access to detection libraries
• Suite of development tools for effective creation and integration of applications with content

The Emotiv EPOC is the world’s first consumer neuroheadset. It detects and processes human conscious thoughts and expressions and non-conscious emotions. By integrating the Emotiv EPOC into their games or other applications, developers can dramatically enhance interactivity, gameplay and player enjoyment by, for example, enabling characters to respond to a player’s smile, laugh or frown; by adjusting the game dynamically in response to player emotions such as frustration or excitement; and enabling players to manipulate objects in a game or even make them disappear using the power of their thoughts.

Headset itself will cost $299 once released into commercial production, the SDK details are available here.

In an interview Dr. Linde described these brains as a form of reincarnation. Over the course of eternity, he said, anything is possible. After some Big Bang in the far future, he said, “it’s possible that you yourself will re-emerge. Eventually you will appear with your table and your computer.” But it’s more likely, he went on, that you will be reincarnated as an isolated brain, without the baggage of stars and galaxies. In terms of probability, he said, “It’s cheaper.”

1. Issue subprime loans to a variety of people.
2. Announce a government-supported plan to bailout existing debtors only if they have poor credit score.
3. Watch the rest of those guys try to decrease their credit score by missing credit card payments.
4. Have the malaises of real estate markets transfer to credit card market, previously unaffected.

Wonder if the Freakonomics blog will pick it up.

The list of available network shows is available here. I watched a few episodes of Family Guy, American Dad, and Simpsons. They do not have a comprehensive collection of all the episodes, but they do have the latest stuff (including episodes that were on TV this past Sunday), and they seem to be filling up their content starting from Season 1. The site is easy to browse, has direct URLs to each episode ( such as http://www.hulu.com/watch/2991/keeping-up-with-the-kardashians-helping-out). The ads are inserted on a regular schedule - before and after the episode, as well as 2 insertions for a 30 minute episode. The commercials I got were those annoying eSurance.com ads, which I tolerate on TV and can tolerate online. All the videos are pausable and viewable later.

I don’t see setting the site up to be my homepage, but as far as number of episodes of good network shows, the quality and selection is generally better than any other video sharing site out there.

Political analysts of the late 80s did not predict the collapse of the Soviet Union, financial markets of the 21st century did not predict Russian default with subsequent sweep on Asian markets, and security analysts did not predict two planes flying into the towers of World Trade Center. However, in the retrospect, we tend to exhibit surprising hindsight, stating that all those events were inevitable, due to [insert a variety of geopolitical arguments here].

Taleb insists that the human mind frequenty:

1. Overestimates the odds of success when applied to oneself. Everybody buying a lottery ticket mentally assumes a much better probability of winning than reality usually presents.
2. Attributes random events in the past as contributing to success/failure. Survivors of the shipwreck would tell the story of how they all prayed for their lives, and thus got saved. While some religious groups would tend to pick up the story as the proof of whatever agenda they’re pitching, nobody generally listens to the stories of those who prayed, but still drowned. Business journalists (with CBS MarketWatch being the worst) frequently abuse this by pitching headlines like Stocks down after Congress increasing military spending in the morning, and Stocks up as Congress approves larger military budget in the afternoon.
3. Looks for order and sequence where it doesn’t exist. Business book writers made a cottage industry out of this by surveying the life stories of prominent individuals, and then reselling those as recipes for success. The quirkier the trait, the better it suits the public. Peter Thiel wakes up early and runs in the morning - I see, so that’s a secret recipe for running a successful hedge fund.
4. Listens to the experts that don’t know any better. There’s a whole bunch of occupations, where expertise is learned, and re-learning it requires significant time investments. Mechanics, doctors, and foreign language interpreters all have such skills. Other kinds of expertise involve looking back at what happened and trying to draw the line of correlation among discordant and random events. Such experts involve financial analysts and government economists.
5. Hungs up on small things without seeing the big picture. If you have significant money in savings accounts, you’re probably busy looking for a better rate. However, the bigger picture, the Black Swan of the US banking industry, is significant asset consolidation among major banks. If one of them gets hit, all of them get hit significantly. The possibility of over-exposure to subprime loans and possible collapse of the banking industry generally escapes the model that your financial analyst presented.

The book is pretty interesting, but hardly coherent. Nevertheless, it’s full of interesting anecdotes and personal observations, that make the book somewhat witty. After reading it I went around reading some other reviews - NYT seemed to not like it, Fool.com did a pretty comprehensive review of the points presented, Slate reviewer seemed to like it. There’s also author’s interview on Colbert report.

Authors take a look at Kazaa, Skype, Alcoholics Anonymous, Craigslist, Wikipedia, eBay, Apache Indian tribes, and Toyota as examples of successful decentralized organizations. Successful in different respects - The eMule Project, covered in the book, doesn’t earn any revenues per se, but manages to sustain all sorts of attempts to exterminate it. Wikipedia is not a money-maker either, but undoubtedly successful as far as the quality of content is concerned. With the Toyota example, the authors point out GM’s Fremont, CA plant as an example of highly centralized, and unsuccessful, project management approach, and compare it to Toyota’s decentralized and highly autonomous approach, where bottom-up innovation is welcomed. The case of the auto plant was apparently covered in the movie Gung Ho, but haven’t seen the movie, I can’t really comment on that.

The authors fill some pages with interviews with Craig Newmark and Jimmy Wales. Valuable content otherwise, it doesn’t really add much to the book - we learn that Jimmy Wales doesn’t do any article editing himself anymore (so they’re even more decentralized than you might have thought), and that Craig Newmark is very user-centric and responds to customer support e-mails. The authors also make a passing remark on the cases when the centralized structure is preferred (U.S. Army, commercial airline, etc.), but don’t really expand on this point, so you’re sold on the idea that decentralized management is applicable to every single company out there.

Sometimes the authors seem to confuse decentralization with just employee empowerment. They quote Jack Welch and current GE as an excellent example of decentralization, while in reality it looks like the corporate structure is still pretty rigid - employees just got motivated to make more decisions themselves, instead of relying on management to tell them what to do.

Overall, the book exposes a good idea, but seems to be hung up on the idea of making the right number of pages to publish a book. Reading this review provides a pretty good overview on what to expect in nine chapters.

From Simons on down, the company encourages openness, whether it’s about market signals that show where a security might be headed or about technology or trading. Frey says he doesn’t recall Simons ever raising his voice at an employee. Simons says new hires are encouraged to troll computer files detailing Renaissance’s past strategies, successful or not. “If Simons’s door was open, you could walk in,” Weinberger says. That would go for everyone from secretaries on up.

What’s also interesting from that description of the company, is there doesn’t seem to be the “corporate way”, or “invented here” syndrome - everyone who just joined the company is invited to question the established beliefs and practices. The fund, nevertheless, is very secretive to the outsiders, but remains hugely profitable, and beats the market even in highly bearish environments.

Pardon me, where’s the restroom? Oh, forget it, where’s more restrooms?

The sign is behind the protective glass, which makes me think it was intentional, or installed by a quite disgruntled [ex-]employee.