Find Jobs
Hire Freelancers

Scraper / Crawler for a website. 400 000 + pages.

$30-250 USD

In corso
Pubblicato più di 5 anni fa

$30-250 USD

Pagato al completamento
Hello everyone, We have a website that we need to gather data from. What we need are user reviews, it is a local p2p shopping platform, something like amazon but much smaller. Link is included in attached file. Framework: Preferably javascript, but python is welcome as well. Data: We are thinking dynamodb local. Mysql is an options too. But we are open to suggestions. Other: We need some kind of proxy implementation, because this website blocks requests after some time. Software needs to run on windows machine. Big part of the project is that it needs to be able to revisit pages every day or so. And update data accordingly. Overall i would like to hear the project from start to finish how it will work. Any ideas out of the box are welcome. Usually don't respond to any automated proposals. Thank you for your time, Kristijan
Rif. progetto: 18369198

Info sul progetto

29 proposte
Progetto a distanza
Attivo 5 anni fa

Hai voglia di guadagnare un po'?

I vantaggi delle offerte su Freelancer

Imposta il tuo budget e le scadenze
Fatti pagare per il lavoro svolto
Delinea la tua proposta
La registrazione e le offerte sui lavori sono gratuite
Assegnato a:
Avatar dell'utente
Hi, I am a Professional Python Developer. I have scraped various websites. Let me describe what i understood about the job. 1. You want to scrape the link provided in the database to get reviews of customers. 2. The code should be automated, i.e. it should run automatically after each day. 3. The site stops bot to scrape so i need to bypass that. 4. Both Dynamodb and mysql are good choices. It operations that you will perform on database decides which one should be used. By the information given here, MySQL sound better. But we need to discuss about the operations on database before i can say MySQL. It won't cost you to just have a chat with me. I am very friendly and good person. I will be glad to help you with this. With Regards, Shubham Sharma
$200 USD in 3 giorni
4,7 (22 valutazioni)
4,6
4,6
29 freelance hanno fatto un'offerta media di $207 USD
Avatar dell'utente
Hi I can implement a desktop application i C# that can run in windows. The data can be exported to Excel or mysql Thanks
$300 USD in 2 giorni
5,0 (119 valutazioni)
7,6
7,6
Avatar dell'utente
Hi employer. i am a expert in web scrapping using python. so if you give me this project, i will satisfy you surely. and i can show previous projects to you. please ping me kindly. thanks.
$222 USD in 3 giorni
4,8 (67 valutazioni)
6,6
6,6
Avatar dell'utente
Hello! I can propose you to make a Python + Scrapy script, that will crawl reviews and save data to csv, json, mysql (on your choice). The script can be put on the cloud and set up to start once a day... It will also use proxies to avoid getting blocked.
$250 USD in 5 giorni
5,0 (38 valutazioni)
5,9
5,9
Avatar dell'utente
Hello, I have read and understood what has to be done in this task. I have sound experience in performing such kind of work as I have done lot of project of similar kind in the past. I think I can assist you in getting this task done with more focus on quality and getting it delivered on time. Regards, Nidhi
$155 USD in 3 giorni
5,0 (22 valutazioni)
5,6
5,6
Avatar dell'utente
Hello Kristijan, I can scrape the data, i have scrapped data from 5)+ wesbites, i have 9+ years of experience, please send me a message so we can discuss more. Thanks!
$155 USD in 3 giorni
4,9 (10 valutazioni)
5,9
5,9
Avatar dell'utente
hello there , i have just checked the website with several requests so i did not see any blocking but maybe after a while can be of course. i will write a script in python with requests and bs4 module. but after testinng the code if there is any blocking i can go for selenium module with proxies. that can take 2 days or maybe shorter i cann not predict at the moment. firstly i have to write the test code and runn it for a while. i can directly connect the output to your mysql databank if u have it. if u dont we can create a table inn local with mysql or sqllite table. we can discuss about it this is the easiest part. so please let me know and we can discuss it further and then i can start to code. cheers Amadeus
$105 USD in 2 giorni
5,0 (18 valutazioni)
5,7
5,7
Avatar dell'utente
Hello. I have checked your project description carefully. Web scrapping is my top skill and i can do it. I hope to work with you. Contact me please. Thank you.
$155 USD in 3 giorni
4,9 (63 valutazioni)
5,7
5,7
Avatar dell'utente
Greetings Kristijan python developer from Serbia here. I can provide you with scraping bot that you need. Zdravo Kristijane Pythonista iz Srbije, mogu da Vam dostavim bota kog zelite. Mladen
$333 USD in 3 giorni
5,0 (29 valutazioni)
5,5
5,5
Avatar dell'utente
Hi. I'm a programmer from Russia with strong skills in Python. This is not an automatic bid, I've read the description and the attachment. Proof: the website is called Kupujemprodajem. I'm good at web-scraping, browser automation and data extraction. I use Grab, Requests, Selenium and BeautifulSoup libraries for this purpose. The fastest solution for your case will be Requests + BeautifulSoup, but if the website has complex anti-scraping protection or complex Javascript, I will need to use Grab or Selenium. I performed complex web scraping project that concerned Amazon and Linkedin, AJAX request and passed anti-scraping tools. I worked a lot with Amazon and its APIs (Product and MWS) and even built scrapers for Amazon. The main problem with that was that Amazon blocked the scrapers if it received too many requests. I solved the problem with proxies or made less requests. About the database: I have basic knowledge of databases and SQL queries that helps me to create applications that manage and store data. Mostly I worked with PostgresQL and SQLite, but I'm also familiar with MySQL and Mongo. I'll be happy to answer your questions. With best regards, Sergey.
$144 USD in 10 giorni
4,9 (13 valutazioni)
5,1
5,1
Avatar dell'utente
I am an excellent web and software developer with over 7+ years experience. I want to use this skills to provide you good works. Please provide more information on your project and get it done. Thanks.
$250 USD in 3 giorni
4,6 (39 valutazioni)
5,2
5,2
Avatar dell'utente
hi there hope you are good we can scrape 200000 pages data we have team work and we can do ti so just ping me.......thanks
$222 USD in 3 giorni
4,9 (73 valutazioni)
4,8
4,8
Avatar dell'utente
I am a Professional Python Scraper I can do Scrapping of the Website with Proxy Enabled I am confident that i can do with high accuarcy Thanks in Advance
$200 USD in 3 giorni
5,0 (70 valutazioni)
4,9
4,9
Avatar dell'utente
I can create a soft that will scrape info you need I can easily integrate it with sql, so results could be saved as csv files or imported direct to database I will create it with UBot Studio, you will get the SOFT only, EXE file To integrate proxy is not a problem but usual free proxy are not good, so you will need to buy proxy what I am offering you for $200: 1. Software, EXE file 2. 1 month free support/update 3. 1 month free us of my private proxies
$200 USD in 10 giorni
5,0 (35 valutazioni)
4,5
4,5
Avatar dell'utente
Hi there, You can choose 2 options, 1) chrome Extension 2) VB Script Advantage of VB Script is that it will run in background, but chrome Extension should have to be operated live. And daily 400,000 is not possible, as each page takes time to load, then only we can fetch data, if you want you can run the same software on different machine or same machine with different indexing. Lets chat to discuss this in details.
$444 USD in 3 giorni
5,0 (10 valutazioni)
3,9
3,9
Avatar dell'utente
Hello, I can develop a Windows desktop application using C# for your project. It will be better and more stable for HTML changes or anti-bot measures. Thanks.
$200 USD in 5 giorni
5,0 (3 valutazioni)
3,5
3,5
Avatar dell'utente
Hi Kristijan, We have extreme experience in Web Scraping. Have done a lot of work of Scraping using python. We use tools scrappy, selenium, scoop for scraping. We scraped data from many e-commerce websites and pushed it in excel and in a database. Also used selenium for creating Wassup bot. Ak
$333 USD in 3 giorni
4,3 (5 valutazioni)
4,0
4,0
Avatar dell'utente
"Hello Dear, Yes, I can work for you as per your project description with amazing exploration and experimental research abilities of mine organization is perfect for that you may require. My work fulfills as well as keep you returning consistently. It is because of the importance behind this field that I am confident & serious when taking these types of projects. Let me introduce with professionalism and skill that exists to supply the best. Regards"
$250 USD in 5 giorni
4,5 (4 valutazioni)
1,7
1,7
Avatar dell'utente
Hello I am very professional in "Entering Types of Information - excel & Data Scraping". You can view employers' comments in this regard. I'm starting now and finishing it. Be sure you will be my regular customer. Send me a message. Maybe we agreed. Thank you for your attention.
$30 USD in 1 giorno
5,0 (1 valutazione)
0,8
0,8
Avatar dell'utente
Hi, is this bid still relevant? I don't think you need a separate proxy implementation as you can use either scrapoxy with your own cloud servers or scrapinghub with crawlera add-on. Whether you're able to use these services or not I'd like to have a look at your code anyway. Regards, Dmitry.
$222 USD in 3 giorni
0,0 (0 valutazioni)
4,1
4,1
Avatar dell'utente
I am doing web scrapping from past 6 months. I have scrapped many sites which includes IMDB websites and my college websites. I do this with beautiful soups.
$77 USD in 3 giorni
0,0 (0 valutazioni)
0,0
0,0

Info sul cliente

Bandiera: SERBIA
Apatin, Serbia
5,0
7
Membro dal gen 11, 2017

Verifica del cliente

Grazie! Ti abbiamo inviato tramite email il link per richiedere il tuo bonus gratuito.
Non è stato possibile inviarti l'email. Riprova per piacere.
di utenti registrati di lavori pubblicati
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Caricamento anteprima
Autorizzazione per la geolocalizzazione concessa.
La tua sessione è scaduta ed è stato effettuato il log out. Accedi nuovamente per piacere.