Data Acquisition, Web Scraping, APIs - Python & MySQL
Project scope
Categories
Information technology Software developmentSkills
beautifulsoup web scraping application programming interface (api) python (programming language) selenium (software) xpath data acquisition scrapy (web crawler) hypertext markup language (html) data retrievalWe at Speedy Snacks would like to improve our back end data acquisition systems. Currently we have to manually copy transaction data from partner websites that is unavailable through an API so we would like to automate the process.
We are looking for students proficient in Python. Some understanding of Scrapy, Scrapy Splash, Selenium, BeautifulSoup, regex, xpath, javascript, CSS, and HTML would also go a long way. They will be tasked with automating the process of copying transaction data from our partners websites and storing it in our database for further processing.
To achieve the project goal student will need to:
- Examine how data is retrieved through partner web portals
- Design a process to automate data retrieval
- Develop a software application to automate data retrieval
I will be available for scheduled weekly meetings to evaluate how the project is going, provide input, and help with any technical challenges. I'll also be available via email, telephone, and most messaging apps to help with any challenges as they arise.
I'll demonstrate and explain our current systems and processes and assist the student with development process.