Ask HN: Beyond Excel, how might a non-technical analyze large amounts of data?

6 pointsareohbe13y ago10 comments

I have to do analysis on a large (in the Excel world) dataset fairly regularly. Ideally, I would like to have some type of repository that I could dump the data into on a monthly basis when I receive the reports and then be able to query and sort the data as needed. Beyond Excel, I'm not sure what the next step up in the world is, but I knew the kind folks here would not be afraid to share an opinion or two.The data consists of multiple performance measurements for multiple unique entities on a per month basis. So currently, my Excel workbook looks something like this:Sheet 1: Summary/Analysis Page that performances basic calculations off subsequent sheetsSheet 2: Performance Metric 1 Column A: Unique Identifier Row A: Months B,B and beyond: DataSheet 3: Performance Metric 2 Column A: Unique Identifier Row A: Months B,B and beyond: DataSheet 4: Performance Metric 3 Column A: Unique Identifier Row A: Months B,B and beyond: Data

10 comments

rjempson13y ago

It depends what you are trying to do exactly, but the excel add-in called powerpivot could be useful if you are trying to aggregate data and do adhoc analysis.

It is free :

http://www.microsoft.com/en-us/bi/powerpivot.aspx

I have analysed lots of BI tools, which could be useful in your scenario, but most require a bit of technical / database knowledge to iron out wrinkles.

ScottWhigham13y ago

+1 for PowerPivot. It kicks butt with large data sets, leverages your existing Excel skills, and extends those skills with DAX.

rpackard13y ago

I worked R & D at a big consumer packaged goods company and ran into a lot of these 'excel messes' especially dealing with raw materials specifications. Here are my suggestions:

The 'R' programming language is pretty useful for statistics and could be used to aggregate a lot of files (and its free). Also if you want to generate dynamic reports using templates then you should look into sweave which is a package in 'R'. (http://www.stat.uni-muenchen.de/~leisch/Sweave/) You could build beautiful PDF reports that you simply run a script to generate if they are easily created into a template. But that would also require learning Latex for creating documents, something that is used heavily in the academic world but I have yet to see it in the business world.

Another approach which I would try is to use a programming language to dump it into a relational database and then use SQL to access the info (which is heavily used in business setting). This option also also allows for others to use it with familiar business tools. My unconventional tools of choice would be Ruby and using Rails to build an interface.

brudgers13y ago

Tabulus might be an option.

http://www.tabulus.com/Home.aspx

deckchain13y ago

I would use Talend for Data Integration to clean and load the data into MySQL. Then write some SQL for the reports that are query-able. Then build Ruby and Rails front-end for the more complicates data views.

infinii13y ago

Try the R programming language

roseleur13y ago

I second that. Actually, Coursera is currently hosting a R-language class at http://www.coursera.org/course/compdata

lsiebert13y ago

R is where you want to go.

calculated_risk13y ago

matlab, stata, c# matlab is by far the best in my opinion

Mz13y ago

GIS software?

Most data has a physical location associated with it (street address, lat-long, something mappable). "A picture is worth a thousand words". Mapping data can help you visualize and mentally process heavy data loads. And Excel is one of the database formats typically compatible with standard software so with a smidgeon of luck you can probably kind of plug and play.

j / k navigate · click thread line to collapse