Find Jobs
Hire Freelancers

Python windows app - hit a list of domains and check sitemap.xml

$250-350 USD

Chiuso
Pubblicato più di 4 anni fa

$250-350 USD

Pagato al completamento
We have a big list of domains we need to check and get an activity score for. To do this our thought was to attempt to get the [login to view URL] file and then parse it to get a last modified date. then we would do some basic math on this to come up with a score. Assuming that the pages are edited in groups of dates i think we can dump in to excel a month and count so we have something like 1/2019 - 10 2/2019 - 37 3/2019 - 0 so in the excel output we have domain | http response | found sitemap | page count from sitemap | months for the last 24.... (make a column for each month for the last 24 months) one thing to make sure you got in the sitemap is that sometimes sitemaps are nested. you will need to follow the link if it leads to another sitemap for that section. for example using google [login to view URL] this main map links to sub maps. if the domain redirects or fails we want to log that I think we can log the http response codes for this so something like 2xx - fine 4xx - failed 5xx - failed 3xx- redirect, then log the redirect name it sends back app should run on windows. and would be cool if we can have config file to put the path to the csv of domains, and maybe some thread count so we can set the performance characteristic of this. We will be running this on some big lists of domains so it might be good to make sure its able to handle running a big list like 10K in some controlled threaded fashion. also we should think about a timeout after a minute or something reasonable so the app doesnt freeze for a dead or bad url? I found this lib which might help you if you needed it - [login to view URL]
Rif. progetto: 22349323

Info sul progetto

20 proposte
Progetto a distanza
Attivo 4 anni fa

Hai voglia di guadagnare un po'?

I vantaggi delle offerte su Freelancer

Imposta il tuo budget e le scadenze
Fatti pagare per il lavoro svolto
Delinea la tua proposta
La registrazione e le offerte sui lavori sono gratuite
20 freelance hanno fatto un'offerta media di $302 USD
Avatar dell'utente
First of all thank you for an excellent description! I can provide you a Scrapy (Python) based web scraping tool that will read a list of domain from a CSV file and allow you to change number of threads to use. It will process sitemaps (including nested ones) and output into another CSV file just like you want. Scrapy has sitemap parser included by default so no need to use ultimate sitemap parser fro your link. Thanks. Roman
$350 USD in 3 giorni
4,9 (638 valutazioni)
8,1
8,1
Avatar dell'utente
Hi. I am ready to write your project Write apps on your demand in many languages (Visual Basic, VBA, VBS, .NET, C#, JS, Python, Java, PowerShell) Write database apps including many db formats: MS Access, MS SQL, SQL Server, MySQL, SQLite, PostgreSQL, Firebird Write Automation apps including: * Automation Desktop apps. Some examples: Automation Playing Games, Automation Start/Stop/Click 3rd party apps * Automation Web apps. Some examples: Automation Web Scraping apps, Automation Web Crawling apps * Automation Data Processing apps: Automation formatting data to a specified template * Automation Data, Document Converting apps * Automation Macro, VBA for all apps in MS Office (Excel, Word, Outlook), OpenOffice, GoogleSheet * Automation Installers/Setups * Convert your Manual tasks to Automation solutions Write Web Service, Web API, Desktop API apps. Some examples: Google API, Bing API, Facebook API, MS API Fix/Solve any errors in your OS, apps
$345 USD in 7 giorni
4,9 (118 valutazioni)
6,5
6,5
Avatar dell'utente
Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON (Scrapy, Selenium) based web scraper as well as WINDOWS BASED web scraping software through which I have crawled many sites such as Craigslist, Amazon, Yelp and many others. I have also worked on complex site to bypass CAPTCHA with the use of PROXY IP bouncing techniques.. Let's work together :) Have a great day! I am glad to see your WORK HISTORY and positive reviews of other freelancers. I am really excited to work with you and would love to have a long-term business association for any of your data related needs less ,,,,,,,  , , ,
$250 USD in 3 giorni
4,9 (120 valutazioni)
6,1
6,1
Avatar dell'utente
I can write python app for Windows with GUI using the mentioned python lib and multi threading . It will save results in csv as per description . I have 6+ experience in Python .
$250 USD in 3 giorni
4,8 (67 valutazioni)
6,0
6,0
Avatar dell'utente
Hi I offer a wide range of services, including , Python , Web Scraping I can create and deliver the project as per the information.I have skilled, expert programmers I'm very excited to assist you in making your Sites successful please feel free to contact me directly to discuss this position further. Thanks
$400 USD in 7 giorni
5,0 (14 valutazioni)
5,5
5,5
Avatar dell'utente
Hi. I am a freelancer working in 24/7 service. I have much experience for 6+ years in Scraping Field, I will produce the output in any format you desire. I think this project is for me and I can finish it for a short time with a quality result. you can also see my profile reviews as my only goal is the client satisfaction, Please son't hesitate to contact me about any question, Kindly initiate chat and let's have a talk. we can start immediately Best Regards.
$250 USD in 2 giorni
4,8 (51 valutazioni)
5,7
5,7
Avatar dell'utente
Hi there, I like that project. Im a Python developer with lots of knowledge in Web Scraping and Python Libraries. I do not exactly understand how you are going to calculate the 'activity score' from sitemaps but Im aware what needs to be done up till, and including, the sitemap parsing part. Let me know more in chat, Pandelis
$280 USD in 7 giorni
4,9 (17 valutazioni)
5,2
5,2
Avatar dell'utente
Hi. 10 k urls or 10 M ?
$277 USD in 3 giorni
5,0 (9 valutazioni)
4,7
4,7
Avatar dell'utente
Hello, dear Customer. I am interested in your project and feel confident after reading your project description. Please contact me so that we can discuss it in more detail. Looking forward to hearing from you soon. Thank you.
$300 USD in 7 giorni
5,0 (13 valutazioni)
4,3
4,3
Avatar dell'utente
Hi, I'm Chunzuo. I read your description and understand what u need. I have already completed python task. Please contact with me to discuss more.
$300 USD in 7 giorni
4,9 (6 valutazioni)
4,0
4,0
Avatar dell'utente
I can start your project immediately. I can provide full-time communication and work your time-zone. If you give me a chance to serve you, I will provide a high quality product within the deadline. Best Regards
$300 USD in 7 giorni
5,0 (4 valutazioni)
2,9
2,9
Avatar dell'utente
Greetings, This project is very similar to one that I recently completed, using Python to test a list of domain names and recording the results of their DNS lookups and HTTP server response status including redirects for over 300K domains. The program used multiple threads to keep the runtime at a reasonable level, with the worker thread count set at runtime. I also have considerable experience using Python to parse HTML and XML files and converting the results to Excel or CSV. After reading your proposal I am confident that I can deliver a robust solution for your project. I am available to start on this project immediately and should be able to deliver the code by Monday. I look forward to working with you!
$300 USD in 3 giorni
5,0 (5 valutazioni)
2,8
2,8
Avatar dell'utente
hi dear. i have read and understood your request.i am interested in your project and can do it very well. I have wide experience in Python development and i am looking forward to contact me, please. i want to consult with you through chatting . thank you
$250 USD in 5 giorni
5,0 (2 valutazioni)
1,3
1,3
Avatar dell'utente
I can do this work
$333 USD in 10 giorni
0,0 (0 valutazioni)
0,0
0,0
Avatar dell'utente
I'm experienced with python, and I am familiar with web design. I look forward to working with you. I plan to finish in six days, but put eight in case life happens. Relevant Skills and Experience I work full time as a systems engineer, so not a programmer specifically, but I have used python quite a bit. Mostly for machine learning and task automation though. Not web skimming.
$250 USD in 8 giorni
0,0 (0 valutazioni)
0,0
0,0
Avatar dell'utente
I can deliver this work in 8 days. I can do initial proof of concept for free in case it needs further confidence building
$266 USD in 8 giorni
0,0 (0 valutazioni)
0,0
0,0

Info sul cliente

Bandiera: UNITED STATES
Austin, United States
4,9
488
Metodo di pagamento verificato
Membro dal mag 9, 2004

Verifica del cliente

Grazie! Ti abbiamo inviato tramite email il link per richiedere il tuo bonus gratuito.
Non è stato possibile inviarti l'email. Riprova per piacere.
di utenti registrati di lavori pubblicati
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Caricamento anteprima
Autorizzazione per la geolocalizzazione concessa.
La tua sessione è scaduta ed è stato effettuato il log out. Accedi nuovamente per piacere.