Scraping for Journalism: A Guide for Collecting Data

Av
Here is the guide in Engligh. An Norwegian version was provided here.

A post from The ProPublica Nerd Blog has the whole story. Here is a part:

Our Dollars for Docs news application lets readers search pharmaceutical company payments to doctors. We’ve written a series of how-to guides explaining how we collected the data.
Most of the techniques are within the ability of the moderately experienced programmer. The most difficult-to-scrape site was actually a previous Adobe Flash incarnationof Eli Lilly’s disclosure site. Lilly has since released their data in PDF format.
These recipes may be most helpful to journalists who are trying to learn programming and already know the basics. If you’re already an experienced programmer, you might learn about a new library or tool you haven’t tried yet.
If you are a complete novice and have no short-term plan to learn how to code, it may still be worth your time to find out about what it takes to gather data by scraping web sites -- so you know what you’re asking for if you end up hiring someone to do the technical work for you.

The tools

With the exception of Adobe Acrobat Pro, all of the tools we discuss in these guides are free and open-source.
Google Refine (formerly known as Freebase Gridworks) – A sophisticated application that makes data cleaning a snap.
Firebug – A Firefox plug-in that adds a host of useful development tools, including the tracking of parameters and files received from web sites you plan to scrape.
Ruby – The programming language we use the most at ProPublica.
Nokogiri – An essential Ruby library for scraping web pages.
Tesseract – Google’s optical character recognition (OCR) tool, useful for turning scanned text into “real,” interpretable text.
Adobe Acrobat – Can (sometimes) convert PDFs to well-structured HTML.
The guides assume some familiarity with your operating system's command-line (Windows, Mac)

Populære innlegg fra denne bloggen

Predicates in Core Data

Av
Database programming is all about filtering and sorting tables linked to each other. According to the book Core Data , «Core Data is Apple’s object graph management and persistency framework for iOS, OS X, watchOS, and tvOS.» In a sense, predicates are an object version of the WHERE clause in SQL. Just like NSFetchRequest represent the FROM clause in SQL. I never have had problems with WHERE and FROM in SQL, but the predicate format strings in Core Data has been quite a struggle for me. I don’t think the struggle is only my fault. Predicates should clearly be thought, at least to me, as a method of writing WHERE clauses. Here is some notes from tutplus’ Core Data and Swift: Relationships and More Fetching : let predicate = NSPredicate(format: "%K == %@", "last", "Doe") The predicate format string uses %K (K stands for Key) for the property name and %@ for the value. let predicate = NSPredicate(format: "%K >= %i", ...
More...

How To Disable Google’s Personalization Of Search Results

Av
Bilde
Do you want to know what the rest of the world gets as answer for their search, instead of what Google thinks you may want to see? I do, therefore the post from MakeUseOf.com by Ann Smarty did catch my interest. She explains it well, both for one-time use and how to change your firefox and chrome browser, and I just copy the one-time part: 2. Disable Google Personalization: One-Time (Browser Independent) Again, you may like the feature and the fact that Google tries to show you what you really want (your favorite sites or sites you are more likely to like on top), but in some cases you will want to see “real” results (for example, to check how your own website is doing). In this case, what you need to do is manually modify the search query URL path: just add this parameter: pws=0 The ‘pws=0′ parameter removes the search personalization: If you don’t want to add the parameter manually, use this browser bookmarklet (courtesy of  Google System Blog ): Turn off Google perso...
More...

InvestorForum by VentureLab in Oslo a success story #startups

Av
Bilde
Photo from Hotel Continental website 200 persons, mostly investors, joined to see what 20 startups could offer in Oslo after the finance crisis . I was astonished to see so many foreign investors, from US and from UK. There was in my opinion a lot of great projects . Since I also presented, I could only follow some: Yes Game : They have ingenuos  reduced the chess board from 8X8 to 5X6 by taking away all second officers. In that way they have create a Chess Attack, which can take you into the final stage of a chess party immediately. They have it for iPhone and very important, also as a physical board. uformia created software very well suited for 3D printing. The CEO Cherie Stamm was an American living in Norway. She softened every Norwegian heart when the said the reason was our beautiful nature. 3D Printing is in my opinion a possibility for the Western society to rebuild the manufacturing industry we lost to China. kikora presented a math editor, which evalua...
More...