Data Wrangling Blog - Latest Comments

Re: Some Datasets Available on the Web

Yuri Burger — Mon, 18 Nov 2013 10:36:52 -0000

one more link: http://endb-consolidated.ai... random 10,000 worldwide companies sampled from HitCompanies (all data in this DB extracted and updated automatically from WWW using AI and Machine Learning): company name and aliases, company description, industry tags, industry codes, registration numbers, addresses, phone numbers, VAT numbers, website, number of about/contact/management/product pages, incorporation date, team size, number of clients and partners, number of emails, number of key changes (client/partner changes, contact changes, people changes), and many more.

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Lukearron — Wed, 21 Sep 2011 18:32:52 -0000

Did anyone solve this problem? I had the same problem, and I downloaded and unpacked the libraries, but perhaps the python library needs to be _placed_ somewhere?

Re: Some Datasets Available on the Web

Modarres Zadeh — Tue, 09 Aug 2011 07:38:02 -0000

Where can i find example datasets for inner workings of an insurance company or bank ?

Re: Some Datasets Available on the Web

Sanjay Kushwaha — Tue, 09 Aug 2011 06:12:39 -0000

can any one help me to get a dataset which can describe the effort estimation of web project including the attributes
TypeProj- Categorical Type of project (new or enhancement).nLang- Ratio Number of different development languages used.DocProc -Categorical If project followed defined and documented process.ProImpr- Categorical If project team involved in a process improvement programme.Metrics -Categorical If project team part of a software metrics programme.DevTeam- Ratio Size of a project’s development team.TeamExp -Ratio Average team experience with the development language(s) employed.TotEffort -Ratio Actual total effort in person hours used to develop an application.EstEffort -Ratio Estimated total effort in person hours to develop an application.Accuracy -Categorical Procedure used to record effort data.

Re: Hidden Video Courses in Math, Science, and Engineering

Alberto — Sat, 30 Jul 2011 21:44:18 -0000

I was aware of many of these, but the UNM Quantum Field Theory is an awesome find. Thanks. I now have a new series of videos to work my way through.

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Easwar — Wed, 13 Jul 2011 10:56:33 -0000

Dear Sir,

I am trying to use MPI for
running Data parallel applications. I read this post and on following up, I have few queries. Would be great, if I can get answers to them.

a) As of now, I am choosing AMI using the browser interface of Amazon. You have uploaded two AMI's one for master (ami-e813f681) and another for slave (ami-eb13f682).
When I launch an instance, I can only choose one. Which one should I
choose ? (Or how should I choose both, in case I need to) ? Do these
AMI's has OPEMPI implementation ?

b) Secondly, if I launch multiple instances of a Large CPU (say 3
instances), I would be get many nodes (say m nodes per one instance, hence i would have m*3 nodes). Can I communicate between nodes
of different instances just as we normally do in MPI ?

c) Are you aware any MPI image with ubuntu ?

d) I am using browser, since I have to connect through proxy. How/Where
do set proxy connections, if I need to start cluster using your python
scripts ?

Easwar

Re: Hidden Video Courses in Math, Science, and Engineering

Marco — Fri, 29 Apr 2011 06:42:57 -0000

Yes you exactly got the point! There are thousands free video lectures out there, but they are mostly at a very elementary level. Hope more advanced stuff will be available in the future.

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Johnliu — Mon, 23 Aug 2010 23:19:59 -0000

Hi,

Does anyone have this running under cygwin in windows?

If so, can you please post your code for ec2-mpi-config.py? I tried using the current file but get lots of errors.

Thanks,
John

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

sumu789 — Thu, 19 Aug 2010 12:00:08 -0000

ok, i figured it out. the connection was being made through my default security group where port 22 was not open.

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Supriyamunshaw — Thu, 19 Aug 2010 10:43:44 -0000

I have sucessfully connected to 5 nodes but am having trouble with the ec2-mpi-config.py script. When I run it, I repeatedly get the following:

---- MPI Cluster Details ----
Numer of nodes = 5
Instance= i-8d2949e7 external_name = ec2-184-72-183-113.compute-... hostname= ip-10-212-239-33.ec2.internal state=
Instance= i-832949e9 external_name = ec2-174-129-138-43.compute-... hostname= domU-12-31-39-10-6C-13.compute-1.internal state=
Instance= i-812949eb external_name = ec2-184-72-141-241.compute-... hostname= domU-12-31-39-0B-00-F8.compute-1.internal state=
Instance= i-872949ed external_name = ec2-174-129-131-171.compute... hostname= domU-12-31-39-09-C4-24.compute-1.internal state=
Instance= i-852949ef external_name = ec2-174-129-61-151.compute-... hostname= domU-12-31-39-0C-D8-57.compute-1.internal state=
5

The master node is ec2-184-72-183-113.compute-...

Writing out mpd.hosts file

scp -i id_rsa-gsg-keypair -o "StrictHostKeyChecking no" id_rsa-gsg-keypair root@ec2-184-72-183-113.compute-1.a...:~/.ssh/id_rsa-gsg-keypair

ssh: connect to host ec2-184-72-183-113.compute-... port 22: Connection timed out
lost connection

ssh -o "StrictHostKeyChecking no" root@ec2-184-72-183-113.compute-1.a... "touch .ssh/authorized_keys"

ssh: connect to host ec2-184-72-183-113.compute-... port 22: Connection timed out

ssh -o "StrictHostKeyChecking no" root@ec2-184-72-183-113.compute-1.a... "cp -r .ssh /home/lamuser/"

It seems I'm having a connection problem. Does anyone what I can do about this?

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Henrywang41 — Tue, 17 Aug 2010 21:51:57 -0000

Can someone please help? mc2-mpi-config.py is giving the following error:

---- MPI Cluster Details ----
Numer of nodes = 2
Instance= i-bf90f5d5 external_name = ec2-184-73-36-216.compute-1... hostname= ip-10-242-1
18-239.ec2.internal state= running
Instance= i-bd90f5d7 external_name = ec2-174-129-77-216.compute-... hostname= ip-10-242-
117-139.ec2.internal state= running

The master node is ec2-184-73-36-216.compute-1...

Writing out mpd.hosts file
Traceback (most recent call last):
File "ec2-mpi-config.py", line 210, in <module>
sys.exit(main())
File "ec2-mpi-config.py", line 65, in main
configure()
File "ec2-mpi-config.py", line 151, in configure
rsakeys = open(homedir + "/.ssh/id_rsa", 'r').read()
IOError: [Errno 2] No such file or directory: 'C:\\Documents and Settings\\Yunzhi Ma/.ssh/id_rsa'

Could this possibly be due to anything about chunk and parsed_response? I printed out parsedresponse:

[['RESERVATION', 'r-30733b5b', '219669225938', 'default'], ['INSTANCE', 'i-bf90f5d5', 'ami-e813f681'
, 'ec2-184-73-36-216.compute-1...', 'ip-10-242-118-239.ec2.internal', 'running'], ['RESER
VATION', 'r-36733b5d', '219669225938', 'default'], ['INSTANCE', 'i-bd90f5d7', 'ami-eb13f682', 'ec2-1
74-129-77-216.compute-1.ama...', 'ip-10-242-117-139.ec2.internal', 'running']]

Thanks so much,
Henry

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Henrywang41 — Mon, 16 Aug 2010 23:56:20 -0000

Hi,

Amazon just recently (last month) released a cloud computing instance (http://developer.amazonwebs...

Does the code and what you describe here work for this new released instance for HPC? (It's CentOS HVM AMI, ami-7ea24a17 under U.S. East)

Thanks,

Henry

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Henrywang41 — Mon, 16 Aug 2010 23:44:05 -0000

Hi,

Is Elasticwulf and MPI essentially the same thing? I'm trying to run some high performance computing using an Amazon EC2 cluster.

Also, is boto necessary to set up a cluster on amazon EC2? What's the difference between boto and Elasticwulf?

Thanks,

Henry

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Henrywang41 — Mon, 16 Aug 2010 22:30:14 -0000

Hi,

What exactly is the difference between Elasticwulf and MPI? Are they the same thing? I'm trying to launch a cluster for HPC, which one is more suitable?

Also, is boto necessary too for launching a cluster?

Thanks,
Henry

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Adeel Jan — Thu, 12 Aug 2010 17:25:53 -0000

I think you need to get python library :)

Adeel Jan.

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

Aarthi 8288 — Sun, 08 Aug 2010 11:32:07 -0000

Hi,

I was trying to launch a cluster using the given scripts. I got the following error :

# ./ec2-start-cluster.py
Traceback (most recent call last):
File "./ec2-start-cluster.py", line 23, in <module>
import EC2
ImportError: No module named EC2

Can anyone help me out ?

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

bearrito — Tue, 03 Aug 2010 20:50:29 -0000

Little late on the thread here but would still like some feedback.

My issues:

1. I was also being prompted for my password. I ended up using the solution that Raghave suggested.

2. I am not seeing /usr/local/src/pyMPI-2.4b2/. That directory doesn't appear to be present. I tried to get around this by copying in fractal.py from my local machine. I end up with the following:

mpirun -np 2 pyMPI /home/lamuser/fractal.py
pyMPI: can't open file '/home/lamuser/fractal.py': [Errno 2] No such file or directory

fractal.py is in that directory.

Advice?

Re: Wikipedia Page Traffic Statistics Dataset

Vinod — Tue, 27 Apr 2010 05:26:16 -0000

Where can i get the number of views for each article. I want to get the views for many articles. Is there any such data set available?

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

miccloud — Mon, 19 Apr 2010 17:57:03 -0000

I have a python script that calls a program installed on the master node and slave nodes.
Can I run it from the master node and get the results with mpirun?

Thanks a lot.

Re: MPI Cluster with Python and Amazon EC2 (part 2 of 3)

miccloud — Sun, 18 Apr 2010 11:27:07 -0000

Can I execute bash script in a node with a job?

Thanks.

Re: Wikipedia Page Traffic Statistics Dataset

tobe — Tue, 09 Mar 2010 13:55:25 -0000

good job!

but how did you get the stats of wikipedia?

greetz, tobe

Re: Some Datasets Available on the Web

Vinoth — Fri, 05 Feb 2010 15:50:47 -0000

hi
can anyone tell me how i can obtain data sets for movie ratings to import into WEKA

Re: Hidden Video Courses in Math, Science, and Engineering

kittipat kampa — Fri, 01 Jan 2010 04:53:23 -0000

Thank you very much for your great effort. Really appreciate!

Re: Some Datasets Available on the Web

manoj — Fri, 01 Jan 2010 01:03:12 -0000

i need Boolean dataset for association mining for my MTech project.Please provide me the address for the same. It will be a great help

Re: Wikipedia Page Traffic Statistics Dataset

Pete Skomoroch — Wed, 23 Dec 2009 14:36:03 -0000

The easiest way would be to access the data from a Linux instance like Ubuntu... With some legwork, you should be able to use Samba somehow to access the volume from Windows - I try to stay away from Windows these days, too many headaches: http://polishlinux.org/linu...