HOW CAN AI BE USED IN WEB SCRAPING  +  Chat GPT ?

Everything changed after year 2023 with the introduction of Chat-GPT.
You can ask it to create code for you that can scrap data from a web site.

Lets start with s simple example.

Step 1.
Download Python and install from here:

Download Python | Python.org

Open your terminal in MAC/Linux or command prompt in Windows and type python :

If you get the above then you are OK.
Type exit() to exit from Python

Step 2.

You need an editor to type or paste the code.
You can use anything like TextEdit in MAC or Notepad++ in Windows, but we stronly suggest to use the free VS Code from here:
Download Visual Studio Code – Mac, Linux, Windows

VS Code will guide you with setup and also suggest to install a Python extension that will assist you.

This is how your code will eventually look like in VS Code:

Step 3.

You will also need Chrome Browser and Google Chrome Driver, you can get the Chrome Driver from here:

Chrome for Testing availability (googlechromelabs.github.io)

Then select the location as shown below and paste it to a new tab to download the zip file (It does not work by just clicking on it, weird !) :

I unziped the file here: C:\temp\chromedriver-win64\

Step 4.

Login to Chat GPT and ask something simple for start:
using python and chrome driver and selenium, i want you to download the source of a web page

Prerequisites

1. Download the ChromeDriver executable from the official website that matches your Chrome version.
2. Install Selenium by running pip install selenium in your terminal.

Type pip install selenium in your terminal to install this library.

GPT came back with some code that did not work 🙂
So to save you time pasting the error back to GPT , we modified the code ourselves:
Paste the code in VS and save it as Sample1.py , for example save it in c:\temp folder.
(make sure double quotes are properly pasted, you might need to retype them in VS Code)