CGI

Draft Version 547 (Thu Dec 1 09:18:39 2005)

The web is far more than just a fancy replacement for inter-library loan
Most of its power comes from the fact that browsers can interact with programs
- More accurately, browsers can ask web servers to interact with programs on their behalf
This lecture looks at what to do if you receive an HTTP request
- Very important that you also attend the next lecture on security before putting programs of your own on the web
Comment on this slide

The Common Gateway Interface

How to let everyday users write programs that handle HTTP requests?
- Option #1: Require them to write socket-level code
  - Complicated and error-prone
  - Can only have one program listening to a socket at a time
- Option #2: have the web server accept the HTTP request, and then invoke the user's code
  - Recompiling the web server every time someone wants to add functionality would be a pain
  - So define a protocol that lets web servers invoke arbitrary external programs
The Common Gateway Interface (CGI) is a simple set of rules that specifies:
- How a web server passes information to a program
- How that program passes information back to the web server
- Yes, it's another protocol…
CGI does not specify:
- A particular language
  - You can use Fortran, the shell, C, Java, Perl, Python…
- How the web server figures out what program to run
  - Each web server has its own rules
  - We'll (briefly) talk about Apache's

When a CGI is run, the web server sends it information:

Through environment variables (which are strings)

Name	Purpose	Example
`REQUEST_METHOD`	What kind of HTTP request is being handled	`GET` or `POST`
`SCRIPT_NAME`	The path to the script that's executing	`/cgi-bin/post_photo.py`
`QUERY_STRING`	The query parameters following `"?"` in the URL	`name=mydog.jpg&expires=never`
`CONTENT_TYPE`	The type of any extra data being sent with the request	`img/jpeg`
`CONTENT_LENGTH`	How much extra data is being sent with the request (in bytes)	`17290`

Table 21.1: CGI Environment Variables

On standard input
- If there is extra data, the web server will send CONTENT_LENGTH bytes to the CGI program on standard input

The CGI program sends data back to the web server via standard output
- In most cases, the web server forwards this directly to the client
- Which means that the CGI program is responsible for creating headers
Comment on this slide

MIME Types

Clients and servers need a way to specify data types to each other
- Remember, bytes are just bytes
Multipurpose Internet Mail Extensions (MIME) standard specifies how to do this
- Organizes data types into families, and provides a two-part name for each type

Family	Specific Type	Describes
Text	`text/html`	Web pages
Image	`image/jpeg`	JPEG-format image
Audio	`audio/x-mp3`	MP3 audio file
`Video`	`video/quicktime`	Apple Quicktime video format
Application-specific data	`application/pdf`	Adobe PDF document

Table 21.2: Common MIME Types

Comment on this slide

Hello, CGI

Simplest possible CGI pays no attention to query parameters or extra data
- Just prints HTML to standard output, to be relayed to the client
- Also prints the Content-type header to tell the client to expect HTML
  - And a blank line to separate the headers from the data
- ```
#!/python/python.exe

# Headers and an extra blank line
print 'Content-type: text/html\n'

# Body
print '<html><body><p>Hello, CGI!</p></body></html>'
```

But the whole point of CGI is to generate content dynamically

Show a list of environment variables

#!/python/python.exe

import os, cgi

# Headers and an extra blank line
print 'Content-type: text/html\n'

# Body
print '<html><body>'
keys = os.environ.keys()
keys.sort()
for k in keys:
    print '<p>%s: %s</p>' % (cgi.escape(k), cgi.escape(os.environ[k]))
print '</body></html>'

You'll use this frequently when debugging…

Comment on this slide

Creating Forms

HTML forms allow users to enter text, choose items from lists, etc.
- Not nearly as sophisticated as desktop interfaces
- Although programmers are doing more every day (particularly using Javascript)
Create a form using a <form/> element
- action is the URL to send data to
- method specifies the type of HTTP request to send (e.g., "POST")

Inside the form, can have:

<select/> elements to let users choose values from a list
- List items specified using <option/> elements
<input/> elements for other kind of data
- If type is "text", get a one-line text entry box
- If type is "checkbox", get an on/off checkbox
- "submit" and "reset" create buttons to submit the form, or re-set the data to initial values

<html>
<body>
<form action="http://www.third-bit.com/cgi-bin/simple_form.py" method="POST">
  <p>Sequence: <input name="sequence" type="text" value="GATTACA"/>
     Search type:
     <select name="search_type">
       <option>Exact match</option>
       <option selected="selected">Similarity match</option>
       <option>Sub-match</option>
     </select>
  </p>
  <p>Programs:
    <input checked="checked" name="program" type="checkbox" value="FROG-11">FROG (version 1.1)</input>
    <input name="program" type="checkbox" value="FROG-beta">FROG (2.0 beta)</input>
    <input checked="checked" name="program" type="checkbox" value="Bayes-Hart">Bayes-Hart</input>
  </p>
  <p>
    <input type="submit"/>
    <input type="reset"/>
  </p>
</form>
</body>
</html>

Figure 21.1: A Simple Form

Notice that each <input/> element has a name attribute
- These become the names of the parameters that the client sends to the server
- The input elements' values are the parameters' values
- Submitting the form shown above with default values produces:
  - os.environ['REQUEST_METHOD']: "POST"
  - os.environ['SCRIPT_NAME']: "/cgi-bin/simple_form.py"
  - os.environ['CONTENT_TYPE']: "application/x-www-form-urlencoded"
  - os.environ['REQUEST_LENGTH']: "80"
  - Standard input: sequence=GATTACA&search_type=Similarity+match&program=FROG-11&program=Bayes-Hart
Comment on this slide

Handling Forms

We could handle form data directly
- Read and parse environment variables
- Read extra data from standard input
The mechanics are the same each time, so use Python's cgi module instead
- Defines a dictionary-like object called FieldStorage
  - Keys are parameter names
  - Values are either strings (if there's a single value assocatied with the parameter) or lists (if there are many)
- When one is created, it reads and stores information contained in the URL and environment
- Program can then read extra data from sys.stdin

Example: show the parameters send to a script

#!/usr/bin/env python
import cgi

print 'Content-type: text/html\n'
print '<html><body>'
form = cgi.FieldStorage()
for key in form.keys():
    value = form.getvalue(key)
    if isinstance(value, list):
        value = '[' + ', '.join(value) + ']'
    print '<p>%s: %s</p>' % (cgi.escape(key), cgi.escape(value))
print '</body></html>'

URL	Value of `a`	Value of `b`
`http://www.third-bit.com/swc/show_params.py?a=0`	`"0"`	None
`http://www.third-bit.com/swc/show_params.py?a=0&b=hello`	`"0"`	`"hello"`
`http://www.third-bit.com/swc/show_params.py?a=0&b=hello&a=22`	`[0, 22]`	`"hello"`

Comment on this slide

Development Tips

When developing scripts, it helps to add import cgitb; cgitb.enable() at the top
- cgitb is the CGI traceback module
- When enabled, it will create a web page showing a stack trace when something goes wrong in your script
Testing whether a FieldStorage value is a string or a list is tedious
- In almost all cases, you'll know whether to expect one value or many
- Use FieldStorage.getfirst(name) to get the unique value
  - Returns the first, if there are many
- FieldStorage.getlist(name) always returns a list of values
  - Empty list if there's no data associated with name
  - If there's only one value, get a single-item list
Comment on this slide

Maintaining State

Often want to change the data a server is managing, as well as read it
- Update a description of an experiment
- Change your preferred email address
- Add items to a shopping cart
Your CGI programs can do this any way they want
- You want it to be simple, fast, and secure…
- …but these are not always compatible
The industrial-strength solution is to use a relational database
- A three-tier architecture (browser, web server, database)
- Figure 21.2: Three Tier Architecture
- CGI program stuffs parameters from HTTP requests into SQL queries
- Runs the queries
- Translates results into HTML to send back to the client
Simple programs can often get away with using files
- The CGI program re-reads the file each time it processes a request
- And re-writes it if there have been any updates

Example: append messages to a web page

Old messages are saved in a file, one per line

Hi, is anyone reading this site?
I was wondering the same thing.
I wasn't sure if we were supposed to post here.
Good point.  Is there way to delete messages?

Script checks the incoming parameters to decide what to do

If newmessage is there, append it, and display results
If newmessage isn't there, someone's visiting the page, rather than submitting the form

#!/usr/bin/env python

import sys, cgi
import cgitb; cgitb.enable()

# Get existing messages.
infile = open('messages.txt', 'r')
lines = [x.rstrip() for x in infile.readlines()]
infile.close()

# Add more data?
form = cgi.FieldStorage()
if form.has_key('newmessage'):
    lines.append(form.getfirst('newmessage'))
    outfile = open('messages.txt', 'w')
    for line in lines:
        print >> outfile, line
    outfile.close()

# Display.
print 'Content-Type: text/html\n'
print '<html><body>'
for line in lines:
    print '<p>' + line + '</p>'
print '''
<form action="http://www.third-bit.com/swc/message_form.py" method="post">
  <p>Your message:
     <input name="newmessage" type="text"/>
  </p>
  <p>
    <input type="submit"/>
    <input type="reset"/>
  </p>
</form>
'''
print '</body></html>'

Note: there is no static HTML page
- What the user sees is always generated by a program
Note also: a lot of this program is devoted to copying values into an HTML template
- There are lots of good systems out there, in many languages, for doing this

Comment on this slide

What About Concurrency?

What happens if two users try to save messages at the same time?
- First instance of message_form.py opens messages.txt, reads lines, closes file
- Second instance opens messages.txt, reads the same lines, closes file
- First instance re-opens file, writes out original data plus one new line
- Second instance re-opens file, writes out original plus a different new line
- First instance's message has been lost!
- A race condition
Solution is to lock the file
- As the name implies, gives one process exclusive rights to the file
- After the first process acquires the lock, any other process that tries to read or write the file is suspended until the first releases it
Mechanics are different on different operating systems
- But the Python Cookbook includes a generic file locking function that works on both Unix and Windows

Modified solution:

Open file
Acquire lock
- If someone else is using the file, blocks until file is available
Read existing messages
If there's a new message:
- Seek to the beginning of the file
- Overwrite data
And oh yeah, release the lock
- Otherwise, everyone else will wait forever

#!/usr/bin/env python

import sys, cgi, fcntl
import cgitb; cgitb.enable()

# Get existing messages.
msgfile = open('messages.txt', 'r+')
fcntl.flock(msgfile.fileno(), fcntl.LOCK_EX)
lines = [x.rstrip() for x in msgfile.readlines()]

# Add more data?
form = cgi.FieldStorage()
if form.has_key('newmessage'):
    lines.append(form.getfirst('newmessage'))
    msgfile.seek(0)
    for line in lines:
        print >> msgfile, line

# Unlock and close.
fcntl.flock(msgfile.fileno(), fcntl.LOCK_UN)
msgfile.close()

# Display.
print 'Content-Type: text/html\n'
print '<html><body>'
for line in lines:
    print '<p>' + line + '</p>'
print '''
<form action="http://www.third-bit.com/swc/locking_form.py" method="post">
  <p>Your message:
     <input name="newmessage" type="text"/>
  </p>
  <p>
    <input type="submit"/>
    <input type="reset"/>
  </p>
</form>
'''
print '</body></html>'

Comment on this slide

Who Are You Again?

What about maintaining state on the client side?
- Need to know which shopping cart to display for a particular user
- But HTTP is a stateless protocol
- If a client makes a second (or third, or fourth…) request, server has no reliable way of connecting it to the first one
  - Can guess based on host, elapsed time, etc.
  - But it's just a guess
Solution is for the server to create a cookie
- A string that is sent to the client in an HTTP response header
- Client saves it (either in memory or on disk)
- The next time the client sends a request to the site, it sends the cookie back to the server
- Like giving someone a claim check for their luggage
To create a cookie:
- Create an instance of Cookie.SimpleCookie
  - Do not use SmartCookie: it is potentially insecure
- Add values to the cookie (it acts like a dictionary)
  - Typically add things like the user's ID, the time of the visit, etc.
- Convert it to a string (e.g., by printing it) to create the required HTTP header
To use a cookie:
- Create a cookie
- Get the value associated with the environment variable "HTTP_COOKIE"
- Pass that value to the cookie's load method

Example: count the number of times a user has visited a web site

If there's no cookie, create one with a count of 1
Otherwise, increment the count
Create a new cookie to send back to the user
Display the count

#!/usr/bin/env python

import sys, os, Cookie
import cgitb; cgitb.enable()

# Get old count.
count = 0
if os.environ.has_key('HTTP_COOKIE'):
    cookie = Cookie.SimpleCookie()
    cookie.load(os.environ['HTTP_COOKIE'])
    if cookie.has_key('count'):
        count = int(cookie['count'].value)

# Create new count.
count += 1
cookie = Cookie.SimpleCookie()
cookie['count'] = count

# Display.
print 'Content-Type: text/html'
print cookie
print
print '<html><body>'
print '<p>Visits: %d</p>' % count
print '</body></html>'

Can control how long a cookie is valid by setting an expiry value
- Either the number of milliseconds
- Or the time it should expire (in UTC)
  - Use time.asctime(time.gmtime())
Warning: do not put sensitive information in cookies
- Browsers store them in files on disk
- Villains can watch network traffic, and steal data
Cookies should instead be random values that act as keys into server-side information
- Talk about this more in the next lecture
Comment on this slide

Summmary

CGI is example of event-driven programming
- The framework invokes your code at specific times, and passes it specific information
- What happens the rest of the time isn't your concern
  - At least, until something goes wrong, and you have to debug it
Simple CGI programs can accomplish a lot
- The entire first generation of web applications were built this way
But please, attend the next lecture on security before making anything publicly available
Comment on this slide

CGI

The Active Web