How to automate web form or API using Bash scripts

How to automate web form or API using Bash scripts

March 01 2023

In this tutorial, I hope to explain where to start when automating web forms or HTTP-based REST APIs using Bash or other UNIX command-line scripting languages. This tutorial should work for bash, the default on most GNU / Linux systems, and zsh, the default for macOS systems.

Step 1: Gathering information about the web form or API

First, we need to know how to use the API. For many APIs, this is a URL, starting with HTTP, like

Questions to ask yourself:

  • Is it an API? Do you know the API? Like, what sort of requests it expects? It typically expects something called a "request" (for our students: relevant lessons include Backend 1.1 and Backend 1.2). Requests can be generated from bash as well, using a Bash program such as curl or wget. We'll learn how to use curl in this tutorial.

  • Is it a web form?

  • If you are trying to automate a web form, when you hit "submit", behind the scenes your browser will send the thing called a "request" to the server. We'll need to simulate this same request using the bash script.

  • You can actually even "freeze" requests sent in the browser into a curl bash command with the following steps:

  1. Bringing up dev tools (Ctrl+J or Command+J)
  2. Click on the network tab
  3. Identify the request that sends the info
  4. Select "Copy as Curl" to copy the info of the request as a curl command

Once you've located the URL, ensure it works by pasting it in your web browser. Do you get JSON or some other data format visible? If so, you probably found the correct API for the resource you want. For example, try clicking on our example API below:

Step 2: Start a new Bash script

  1. Create a new Bash script by opening an empty file in your preferred text editor, such as the free open VS Code by Microsoft, that should work on all platforms (Windows, Mac and Linux). For now, write echo Hello API world into it
  2. Save the Bash script using a filename, such as (avoid using spaces in filenames, consider all-lowercase)
  3. Navigate to where you saved the Bash script in a Terminal. You know you are in the right place when you see the filenmae (e.g. when you run ls (i.e., this means you are in the same directory as the thing). (In Linux, you can right-click on a window and select "Open in Terminal". In other OS's, open your terminal, type cd followed by space, then drag over the directory)
  4. Run the bash script with the command bash

All together, it's like this:

$ cd /home/michaelb/projects/myapiproj/
$ ls
$ bash
Hello API World

Step 3: Add in the cURL commands to get data from the API

The cURL command copied from the browser following the steps above, and then can be pasted into a bash script to repeat that exact request. For example:

echo Modulo info

echo Cheatsheet info

Step 4: Possibly add in form data as POST data or JSON APIs

The cURL command can do a lot, such as using POST and also including "post data", which would be the form data itself. If you use the "Copy as cURL" command then you get this automatically. Otherwise, you'll need to use form encoding, as such:

curl -d "param1=value1&param2=value2" -X POST

curl -d '{"param1":"value1", "param2":"value2"}' -H "Content-Type: application/json" -X POST

Want to learn more?

Continue your coding journey with Kickstart Coding's unique online program.

Apply now

Leave this field blank Don't touch this one either

  Free Sample

Apply now and "audit" a video class for free!

Kickstart Fundamentals - 1.1 - HTML