INTERN

INTERN get html html parse BeautifulSoup HOW WOULD YOU DESIGN AND BUILD A WEB SCRAPER INTERN OKAY WELL A SIMPLE PYTHON SCRIPT WE'D FIRST GET THE RAW HTML USING REQUESTS THEN WE'D USE BEAUTIFUL SOUP TO PARSE THAT HTML AND EXTRACT WHATEVER INFORMATION WE NEED JR DEV inspect requests in network tab example.com Python call underlying API directly with copied info some API UP THE WEBSITE IN CHROME OPENING UP DEVELOPER TOOLS THE NETWORK TAB TO SEE WHAT INFORMATION THAT WEBSITE IS ACTUALLY GETTING ON THE BACK END THEN I'D COPY THOSE API REQUESTS MYSELF INFORMATION DIRECTLY FROM THEIR API WITHOUT HAVING TO PARSE OF HTML AND JAVASCRIPT SENIOR ENGINEER robots.txt retries backoff alerts Database VM site1.com site2.com site3.com I'D START BY READING DOT TXT TO MAKE SURE THE WEBSITE ALLOWS SCRAPING AND IF IT DOESN'T OFFICIAL API THEN WE'LL BUILD A SIMPLE SYSTEM WE CAN USE TO SCRAPE ANY WEBSITE EVERY 24 HOURS WE'LL USE PLAYWRIGHT TO SCRAPE DATA FROM EVERY SINGLE SITE THINGS LIKE READ HTML BUT ALSO PRESS BUTTONS LOGIN RENDER JAVASCRIPT ALL THAT FUN STUFF FROM A SITE AND WRITE IT TO OUR DATABASE TO ACCESS THAT DATA AGAIN AND AGAIN TO OUR DATABASE INSTEAD OF HAVING TO THIS REDUCES OUR CHANCES OF GETTING BANNED AND OF COURSE FOR TO EXTERNAL WEBSITES WILL HAVE THINGS LIKE RETRIES EXPONENTIAL BACKOFF AND ALERTS IF SOMETHING ON THE WEBSITE CHANGES AND WE NEED TO UPDATE THE SCRAPER