Subscribe to DSC Newsletter
  • Male
  • Chennai,TamilNadu
  • India
Share on Facebook

Gifts Received


Pandiyarajan has not received any gifts yet

Give a Gift


Pandiyarajan's Page

Profile Information

Short Bio:
Working in Sciera,As Perl Developer.
Sciera Solution Pvt Ltd.
Big Data
Finding a new position, Networking, New venture

Pandiyarajan's Blog

Need solution To Crawl 200 million Web Page in 10-1 month Using Perl and hadoop is this possible?

Posted on November 21, 2013 at 10:10pm 1 Comment

Hi all Greetings from Prajan,

 I am Prajan(Pandiyarajan), Working As a Perl Developer in Sciera Solution.I have 2+ year in perl and Big Data .

 I got a difficult task from my manager ,that was i need to Crawl 200 million URL in same Domain with in 20-1 month time ,i have tried in Perl i got Maximum 250 hits only in 60 seconds.I heard will make this using Hadoop But i don't know hadoop ,Any one can You give a solution for my task with detail then how can i reach…


Comment Wall

You need to be a member of Data Science Central to add comments!

Join Data Science Central

  • No comments yet!

© 2021   TechTarget, Inc.   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service