Data Wrangling Blog - Latest Comments in Amazon Elastic MapReduce: A Web Service API for Hadoop

Re: Amazon Elastic MapReduce: A Web Service API for Hadoop

Peter Skomoroch — Sat, 11 Apr 2009 20:28:17 -0000

Peter T,

Good question, I think this is something Amazon needs to sort out in general. For elastic mapreduce (EMR) it is easier to work with chunked S3 files at the moment, so I just run a script to copy the raw data up to s3 in chunks using s3cmd. Outside of EMR I usually use single EBS volumes for datasets < 1 TB, since they can be directly mounted on EC2 and are easier to work with. I haven't tried the EBS + EMR hack yet, but will let you know how that goes if I try it.

Contact me with the exact data sizes you are dealing with and maybe I can suggest an approach...

-Pete

Re: Amazon Elastic MapReduce: A Web Service API for Hadoop

Peter Tonellato — Sat, 11 Apr 2009 19:55:13 -0000

Peter,

Great stuff - thanks very much for posting this.

We ahve 100GB-multi TB input files required for various none-java analysis applications. Would you recommend we use the same approach so beautifully presented above and in the python tutorial - or do you recommend we take a different approach to making the input data available?

Peter

Re: Amazon Elastic MapReduce: A Web Service API for Hadoop

Peter Skomoroch — Fri, 03 Apr 2009 21:06:32 -0000

Here's an idea for a streaming hack I mentioned on the forums. There is no built in support for pulling data from EBS volumes instead of S3 and boto isn't installed, but you you could use the distributed cache to install boto, mount an EBS public dataset from within the first mapper, and load it onto hdfs. Something like this:

1) send a zipped boto src directory, renamed with a .mod extension over the distributed cache

2) Add an intial dummy JobFlow step where the mapper opens boto.mod, reads in your credentials, imports boto, and mounta the EBS snapshot using your AWS credentials

import zipimport
importer = zipimport.zipimporter('boto.mod')
boto = importer.load_module('boto')

mount public dataset via boto:

ec2-attach-volume vol-e84aae81 -i i-a09104c9 -d /dev/sdf
mkdir /mnt/genbank
mount /dev/sdf /mnt/genbank

3) after the volume is mounted, shell out within the mapper and copy the data from the EBS volume to HDFS:

hadoop fs -put /mnt/genbank /home/hadoop/genbank

4) proceed with the next JobFlow step, using hdfs:///home/hadoop/genbank as input

Re: Amazon Elastic MapReduce: A Web Service API for Hadoop

Peter Skomoroch — Fri, 03 Apr 2009 10:53:48 -0000

Thanks Steve, I've been following the Mahout mailing list for a while and look forward to seeing more machine learning algorithms implemented in mapreduce. It is a young project, but I'm interested to see how it develops.

This post was extremely short since I wanted to get the announcement out quickly, I'll have a more to post about machine learning with Hadoop soon. Until then, check out http://delicious.com/pskomo... and http://delicious.com/pskomo...

Re: Amazon Elastic MapReduce: A Web Service API for Hadoop

Steve Lianoglou — Fri, 03 Apr 2009 10:00:40 -0000

Being that this blog is related to machine learning as well, I think some of your readers would be interested to know about the mahout project[1], which is an apache project whose aim is to develop various machine learning algorithms in the mapreduce paradigm, using hadoop.

[1] mahout: http://lucene.apache.org/ma...