How to automate form submissions/logins

Python has a lot of libraries to manipulate the web.

For example: If you want any web browser automation, there’s Python and Selenium. While if you want to scrape web content, there’s Python and Websoup.

These tools are helpful for web developers to simplify our lives.

In this tutorial, we’ll be covering how to automate submitting forms and logging into sites.

I build courses. One important function is that students should be able to go from the beginning of the course, to the end of the course, without hiccups. The courses I’ve built have roughly 60-80 web pages. The biggest course was almost 200 web pages.

In order to do check to see if a student can go from the beginning of a course to the end, a test would need to log in, and manually click on every SINGLE lesson, and move through it. With 80 web pages and at 0.5 minutes a page, that’s 40 minutes. That’s not including homeworks and special user flags that shows only if certain criteria happens.

Other use cases may be:

  • You want to submit 50 forms to your form software to make sure it’s grabbing the data correctly.
  • You want bots to grab all links on every single page.
  • You want to quickly create 50 test users.

What you will learn:

You’ll be using Python 3 and the Selenium library.

The overall goal is to first install Python3, then Selenium using PIP. Installing Python also installs IDLE, which allows you to write Python code and execute it.

That’s a lot of random terms I just threw I at you. Let me break it down.

  • Python (or Python3): The current version of Python as of this writing is Python 3.7. You should get that, or higher versions, if possible.
  • Selenium: This is a library that allows us to do a lot of web automation.
    • Library – you ask? A library is a bunch of scripts to make your programming life easier. For example: The math library. You can totally write your own versions of say, figuring out the PI. But why do that when someone else already has done it? A team of developers wrote the Selenium Library that does all sorts of cool things, so we don’t have to think about much of the heavy lifting.
  • PIP: PIP is a recursive acronym that stands for “PIP Installs Packages” or “Preferred Installer Program”. It’s what we use to install other libraries.
  • IDLE: This is the development environment that we will use for Python. It’s super basic, and great for beginners. As your code gets more complex, you’ll use something else. But for this exercise – stick with IDLE.

If you have no idea where to start – I found the best tutorials that lay it out as clearly as possible.

#1 – Download Python & PIP:

I know what what you’re asking. How do we get Python?

Fortunately – most computers already have Python 2 installed. What we WANT is Python 3!

But let’s first get PIP.

Sidenote: Why is there a Python 2, and a Python 3? The Python community moves slowly to implement changes. Python 2 is legacy code, and will not be maintained after 2020. So unless you somehow get a job where you manage Python 2 code (which is highly unlikely), focus on Python 3. To add salt to the wound, someone made a countdown clock of Python 2’s death here.

#2 – Test to see if Python is properly installed:

If Python isn’t installed correctly, there’s no point in moving forward. In an effort to measure twice, cut once – we will be testing to see if things are working correctly.

If you see a response like

Python 3.7.0

It means that Python is installed. The version may be higher.

For Macs, in your Terminal, type:

python3 --version

For PCs, you will need to have installed Python AND Adjust System Variables to ensure you can modify it from the command line.

Open up Command Line or Power Editor, and type:

python --version

#3 – Installing Selenium and the various Webdrivers

We installed Python and PIP. Now to get Selenium.

To get Selenium, use this cheat code.

pip install selenium

As a reminder, Selenium is the Python library we use for web automation. Selenium is just the library, and developed API so third-party authors can develop the webdrivers to allow it to talk to browsers.

That way, the Selenium team can focus on their automation, while another team can focus on how to get Selenium to work on their browser platform.

For example:

  • The Chromium team made their own webdriver for Selenium called chromedriver.
  • The Firefox team made their own webdriver for Selenium called geckodriver.
  • The Opera team made their own webdriver for Selenium called operadriver.

In our code, you’ll see me do things like “Open Firefox”, and “pass this link to Firefox”, and finally “Close Firefox”. I would need the geckodriver to do that.

To install those web drivers, use this cheat code. In your command line, type:

pip install webdriverdownloader

#4 – Using the IDLE and writing Python code:

To make this simple, we’ll be writing and running our Python 3 code in IDLE. What is IDLE?

IDLE (short for integrated development environment[1][2] or integrated development and learning environment[3]) is an integrated development environment for Python, which has been bundled with the default implementation of the language since 1.5.2b1.[4][5] It is packaged as an optional part of the Python packaging with many Linux distributions. It is completely written in Python and the Tkinter GUI toolkit (wrapper functions for Tcl/Tk).


IDLE is intended to be a simple IDE and suitable for beginners, especially in an educational environment. To that end, it is cross-platform, and avoids feature clutter.

via Wikipedia

Code structure: Python is that it uses INDENTATION to segment code blocks. Other popular programming languages (Like Javascript, C++, and PHP) uses curly brackets. How you indent the code matters!

Python also doesn’t end statements with semicolins. You can still do it, but there’s no reason to do so.

# WRITTEN IN PYTHON
def print_jokes(): 
  print("The problem with kleptomaniacs is that they always take things literally.")

# WRITTEN IN PHP
function print_jokes() {
  echo "The problem with kleptomaniacs is that they always take things literally.";
}

#5 – Running a log-in script:

Below, I have a log in script that allows me to log into

## Selenium Log In Test
## 
## Description: This code logs into https://imgur.com automatically using Firefox.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time

## Define which browser to use
driver = webdriver.Firefox()
driver.implicitly_wait(5)
    ## implicity_wait makes the bot wait 5 seconds before every action
    ## so the site content can load up

## Define the user and email combo. 
bot_name = 'testingwithselenium'
bot_email = bot_name + '@gmail.com'
bot_password = 'testingwithselenium1' ## required a number, so 1 is at the end
print(bot_name + " is getting started")

## Open the login page
driver.get('https://imgur.com/signin')

## This code checks the page for specific HTML tags
## This is looking for the element with ID 'username'			
textfield_username = driver.find_element_by_id('username')
textfield_username.clear()
textfield_username.send_keys(bot_email)

## This is looking for the element with ID 'password'			
textfield_email = driver.find_element_by_id('password')
textfield_email.clear()
textfield_email.send_keys(bot_password)

## This is looking for the element with the class 'btn_action'			
submit_button = driver.find_element_by_class_name('btn-action')
submit_button.click()

## wait for 2 seconds
time.sleep(2)

## RESULT == it should log you in

## Now to close the browser
try:
    print(bot_name + "closing")
finally:
   driver.quit()
    

print("task complete")

Additional information

That script is the barebones version of logging in.

If you had to make 50 test accounts, look into turning that script into a function, and then running a loop.

Conclusion

There’s a lot of use cases for automation. It takes a while to set up. But once you do, there’s a lot of directions you can go into.

Leave a Comment