Hadoop: What it is, how it works, and what it can do

Av
A post from O'Reilly Radar - Insight, analysis, and research about emerging by James Turner has the whole story. Here is a part:

Hadoop gets a lot of buzz these days in database and content management circles, but many people in the industry still don't really know what it is and or how it can be best applied.
Cloudera CEO and Strata speaker Mike Olson, whose company offers an enterprise distribution of Hadoop and contributes to the project, discusses Hadoop's background and its applications in the following interview.

Where did Hadoop come from?


Mike OlsonMike Olson: The underlying technology was invented by Google back in their earlier days so they could usefully index all the rich textural and structural information they were collecting, and then present meaningful and actionable results to users. There was nothing on the market that would let them do that, so they built their own platform. Google's innovations were incorporated intoNutch, an open source project, and Hadoop was later spun-off from that. Yahoo has played a key role developing Hadoop for enterprise applications.


What problems can Hadoop solve?


Mike Olson: The Hadoop platform was designed to solve problems where you have a lot of data — perhaps a mixture of complex and structured data — and it doesn't fit nicely into tables. It's for situations where you want to run analytics that are deep and computationally extensive, like clustering and targeting. That's exactly what Google was doing when it was indexing the web and examining user behavior to improve performance algorithms.
Hadoop applies to a bunch of markets. In finance, if you want to do accurate portfolio evaluation and risk analysis, you can build sophisticated models that are hard to jam into a database engine. But Hadoop can handle it. In online retail, if you want to deliver better search answers to your customers so they're more likely to buy the thing you show them, that sort of problem is well addressed by the platform Google built. Those are just a few examples.

Strata: Making Data Work, being held Feb. 1-3, 2011 in Santa Clara, Calif., will focus on the business and practice of data. The conference will provide three days of training, breakout sessions, and plenary discussions -- along with an Executive Summit, a Sponsor Pavilion, and other events showcasing the new data ecosystem.

Save 30% off registration with the code STR11RAD

How is Hadoop architected?


Mike Olson: Hadoop is designed to run on a large number of machines that don't share any memory or disks. That means you can buy a whole bunch of commodity servers, slap them in a rack, and run the Hadoop software on each one. When you want to load all of your organization's data into Hadoop, what the software does is bust that data into pieces that it then spreads across your different servers. There's no one place where you go to talk to all of your data; Hadoop keeps track of where the data resides. And because there are multiple copy stores, data stored on a server that goes offline or dies can be automatically replicated from a known good copy.
In a centralized database system, you've got one big disk connected to four or eight or 16 big processors. But that is as much horsepower as you can bring to bear. In a Hadoop cluster, every one of those servers has two or four or eight CPUs. You can run your indexing job by sending your code to each of the dozens of servers in your cluster, and each server operates on its own little piece of the data. Results are then delivered back to you in a unified whole. That's MapReduce: you map the operation out to all of those servers and then you reduce the results back into a single result set.
Architecturally, the reason you're able to deal with lots of data is because Hadoop spreads it out. And the reason you're able to ask complicated computational questions is because you've got all of these processors, working in parallel, harnessed together.

Populære innlegg fra denne bloggen

Predicates in Core Data

Av
Database programming is all about filtering and sorting tables linked to each other. According to the book Core Data , «Core Data is Apple’s object graph management and persistency framework for iOS, OS X, watchOS, and tvOS.» In a sense, predicates are an object version of the WHERE clause in SQL. Just like NSFetchRequest represent the FROM clause in SQL. I never have had problems with WHERE and FROM in SQL, but the predicate format strings in Core Data has been quite a struggle for me. I don’t think the struggle is only my fault. Predicates should clearly be thought, at least to me, as a method of writing WHERE clauses. Here is some notes from tutplus’ Core Data and Swift: Relationships and More Fetching : let predicate = NSPredicate(format: "%K == %@", "last", "Doe") The predicate format string uses %K (K stands for Key) for the property name and %@ for the value. let predicate = NSPredicate(format: "%K >= %i", ...
More...

How To Disable Google’s Personalization Of Search Results

Av
Bilde
Do you want to know what the rest of the world gets as answer for their search, instead of what Google thinks you may want to see? I do, therefore the post from MakeUseOf.com by Ann Smarty did catch my interest. She explains it well, both for one-time use and how to change your firefox and chrome browser, and I just copy the one-time part: 2. Disable Google Personalization: One-Time (Browser Independent) Again, you may like the feature and the fact that Google tries to show you what you really want (your favorite sites or sites you are more likely to like on top), but in some cases you will want to see “real” results (for example, to check how your own website is doing). In this case, what you need to do is manually modify the search query URL path: just add this parameter: pws=0 The ‘pws=0′ parameter removes the search personalization: If you don’t want to add the parameter manually, use this browser bookmarklet (courtesy of  Google System Blog ): Turn off Google perso...
More...

InvestorForum by VentureLab in Oslo a success story #startups

Av
Bilde
Photo from Hotel Continental website 200 persons, mostly investors, joined to see what 20 startups could offer in Oslo after the finance crisis . I was astonished to see so many foreign investors, from US and from UK. There was in my opinion a lot of great projects . Since I also presented, I could only follow some: Yes Game : They have ingenuos  reduced the chess board from 8X8 to 5X6 by taking away all second officers. In that way they have create a Chess Attack, which can take you into the final stage of a chess party immediately. They have it for iPhone and very important, also as a physical board. uformia created software very well suited for 3D printing. The CEO Cherie Stamm was an American living in Norway. She softened every Norwegian heart when the said the reason was our beautiful nature. 3D Printing is in my opinion a possibility for the Western society to rebuild the manufacturing industry we lost to China. kikora presented a math editor, which evalua...
More...