Client-Side Web Programming

Small Pieces, Loosely Joined

• The Unix command line was the world's first component object model
  • Allowed programmers to build small pieces, then connect them in arbitrary ways
• Key features:
  • Low barriers to entry: it's easy to add one more tool to the toolbox
  • Common data format: stream of strings
  • Common communication protocol: stdin, stdout, and zero/nonzero exit codes
• The Web succeeded (in part) because it followed the same model
  • Data format: HTML (now XML)
  • Communication protocol: HTTP
• This lecture looks at how to build programs that get data from the web
  • Next lecture will discuss simple ways to build programs that supply data
  • If you want to know more, [Goerzen 2004] is a comprehensive guide
• Comment on this slide

Distributed Is Different

• Distributed systems are fundamentally different from unitary systems
  • Small program can ignore these differences…
  • …but every industrial-strength application eventually has to deal with them
• Difference #1: concurrency
  • As in databases, means “several things happening at once”
  • Can lead to:
    • Deadlock: A is waiting for B while B is waiting for A
    • Race conditions: final result depends on whether A or B goes last
• Difference #2: partial failure
  • One component fails while others are still healthy
  • If you've waited five seconds for a web site to respond, should you assume that it's down, or keep waiting?
  • There's a whole industry devoted to dealing with this
• Both differences make distributed applications much harder to debug than unitary ones
  • Often have heisenbugs (which only appear intermittently)
  • And it's usually impossible to get a complete picture of the system's state
  • Solution: get it right the first time
• For now, we will ignore both problems
  • Write small programs that use the web, but aren't used by others
  • Assume that everything else is reliable
• Comment on this slide

Under the Hood

• These days, the Internet runs on a family of communication standards called Internet Protocol (IP)
  • Defines everything from electrical signal strengths on up
• Using IP, process communicate through sockets
  • Each socket is one end of a point-to-point communication channel
    • Provides the same kind of read and write operations as files
  • Every socket is uniquely identified by host address and a port number
  • A host address is four 8-bit numbers, like 24.153.22.195
    • Most addresses have symbolic names, like www.third-bit.com
  • A port is just a number in the range 0-65535
    • 0-1023 are reserved for the operating system
• Most commonly used member of the IP family is Transmission Control Protocol (TCP)
  • Guarantees that any data sent is received, in order
    • Like a phone call
  • Lots of applications use TCP directly
    • Email, instant messaging, FTP, games…
  • But don't worry: we'll be programming at a much higher level
• Comment on this slide

The Hypertext Transfer Protocol

• HTTP (the HyperText Transfer Protocol) specifies how programs exchange documents over the web
  • Originally designed to support file download
  • Has grown in many ways
    • Some of them unnatural
• Three stages in a typical transaction:

  • Figure 20.1: HTTP Request Cycle

  • A client program opens a connection to a web server
    • Clients are typically browsers, such as Firefox
    • Apache Web Server is the most widely used, but many others exist
  • The client sends a request containing
    • The thing the client wants (such as the name of a web page)
    • Headers that define the type of transaction
    • Anything else the server might need (e.g., a file for upload)
  • The server replies with:
    • Headers that define the status (success or failure) and the type of data being sent
    • Data (such as the actual web page)
• This cycle may be repeated many times to display one web page
  • Each image is requested and downloaded separately
• Comment on this slide

HTTP Request

• An HTTP request has three parts

  • Figure 20.2: HTTP Request Format

• HTTP request line
  • HTTP method is usually either GET or POST
    • GET typically used to fetch information
    • POST usually used to submit form data, or to upload files
  • URL is the thing the request wants
    • Typically a path to a file, such as /index.html
    • But other information (discussed below) can be put there as well
    • Entirely up to the server how to interpret the URL
  • HTTP version
    • Usually HTTP/1.0
• Headers
  • A set of name/value pairs
    • A given name may appear many times, i.e., may be associated with many values
  • Examples:
    • Accept: text/html
    • Accept-Language: en, fr
    • If-Modified-Since: 16-May-2005
• Body is any extra data associated with the request
  • Used with web forms, to upload files, etc.
• Note: there's no magic
  • An HTTP request is just text—any program that wants to can create them or parse them
• Comment on this slide

HTTP Response

• Just as simple as a request

  • Figure 20.3: HTTP Response Format

• HTTP response line
  • Once again, version is typically HTTP/1.0
  • Status code is a number indicating what happened
    • 200: everything worked
    • 404: page not found
  • Status phrase repeats that information in a human-readable phrase (like “OK” or “not found”)
• Headers are once again name/value pairs
• Then there must be a blank line to signal the end of headers
  • Common error to forget this
• Body is whatever data the server wants to send back
  • A web page, an image, etc.
• Comment on this slide

Example

• Fetch a page from the course site

  • import sys, socket
    
    HttpRequest = \
    '''GET /greeting.html HTTP/1.0
    
    '''
    
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect(("www.third-bit.com", 80))
    
    s.send(HttpRequest)
    
    response = ''
    while True:
        data = s.recv(1024)
        if not data:
            break
        response += data
    s.close()
    
    print response
    
    

    HTTP/1.1 404 Not Found
    Date: Sat, 02 Jul 2005 11:16:44 GMT
    Server: Apache/2.0.54 (Debian GNU/Linux) DAV/2 SVN/1.1.4 mod_python/3.1.3 Python/2.3.5 PHP/4.3.10-9 mod_ssl/2.0.54 OpenSSL/0.9.7e
    Content-Length: 394
    Connection: close
    Content-Type: text/html; charset=iso-8859-1
    
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    <title>404 Not Found</title>
    </head><body>
    <h1>Not Found</h1>
    <p>The requested URL /greeting.html was not found on this server.</p>
    <hr>
    <address>Apache/2.0.54 (Debian GNU/Linux) DAV/2 SVN/1.1.4 mod_python/3.1.3 Python/2.3.5 PHP/4.3.10-9 mod_ssl/2.0.54 OpenSSL/0.9.7e Server at pyre.third-bit.com Port 80</address>
    </body></html>
    
    

• Request has no headers
  • So the blank line that signals …end of headers” is right after the request line
• Create a socket
  • socket.AF_INET means “an Internet socket”
  • socket.SOCK_STREAM means “use TCP”
• Connect to the server
  • Port 80 is the standard one for web servers
• Send the request, and read response data until there's nothing more to read
• Comment on this slide

Fetching Pages

• Opening sockets, constructing HTTP requests, and parsing responses is tedious
  • So most languages provide libraries to do the work for you
  • In Python, that library is called urllib
• urllib.urlopen(URL) does what your browser would do if you gave it the URL
  • Parse it to figure out what server to connect to
  • Connect to that server
  • Send an HTTP request
  • Give the calling program an object that looks like a file, from which to read response data
• Example: read the test page the easy way

  • import urllib
    
    instream = urllib.urlopen("http://www.third-bit.com/greeting.html")
    lines = instream.readlines()
    instream.close()
    for line in lines[:5]:
        print line,
    

• Note: readlines wouldn't do the right thing if the thing being read was an image
  • Use read to grab the bytes in that case
  • Up to the client application to do the right thing
• Example: make a list of the links in a web page
  • The first step in build a web spider that can explore the web on its own
    • That, plus a search engine, and you're Google…
  • Fetch the page
  • Parse it to extract all the links
    • Should use XML, but regular expressions are good enough to give you the idea

  • import urllib, re
    from sets import Set
    
    instream = urllib.urlopen("http://www.third-bit.com/index.html")
    page = instream.read()
    instream.close()
    
    links = re.findall(r'href=\"[^\"]+\"', page)
    temp = Set()
    for x in links:
        temp.add(x[6:-1])
    links = list(temp)
    links.sort()
    for x in links:
        print x
    

    #ack
    #news
    #rules
    #search
    /MT/mt.cgi
    /blog
    /cgi-bin/awstats.pl?config=www.third-bit.com
    http://books.slashdot.org/books/05/06/16/1930227.shtml
    http://projects.edgewall.com/trac
    http://subversion.tigris.org
    http://weblog.infoworld.com/udell
    http://weblog.infoworld.com/udell/2005/06/23.html
    http://www.amazon.com/exec/obidos/redirect?path=ASIN/0974514071&link_code=as2&camp=1789&tag=thirdbit-20&creative=9325
    http://www.amazon.com/exec/obidos/redirect?tag=thirdbit-20&creative=9325&camp=1789&link_code=as2&path=ASIN/0974514071
    http://www.apache.org
    http://www.artima.com
    http://www.artima.com/chapters/book.jsp?num=114451
    http://www.cs.utoronto.ca
    http://www.debian.org
    http://www.jonahgroup.com
    http://www.list.org
    http://www.oreillynet.com
    http://www.oreillynet.com/cs/user/print/a/5862
    http://www.pragmaticprogrammer.com/titles/gwd/index.html
    http://www.provost.utoronto.ca
    http://www.python.org
    http://www.python.org/psf
    http://www.sf.net
    http://www.techbookreport.com/tbr0172.html
    http://www.third-bit.com/pyweb/index.html
    http://www.third-bit.com/swc
    http://www.third-bit.com/trac/argon
    http://www.third-bit.com/trac/pyre
    http://www.third-bit.com/trac/pyre/newticket
    http://www.third-bit.com/trac/swc
    http://www.third-bit.com/~gvwilson
    http://www.utoronto.ca
    mailinglists.html
    mailto:gvwilson@cs.utoronto.ca
    old-presentations.html
    past.html
    reading.html
    style.css
    

• Comment on this slide

Passing Parameters

• Sometimes want to provide extra information as part of a URL
  • Example: when searching on Google, have to specify what the search terms are
• Could do this as part of the URL
  • Amazon puts ISBNs in URLs
• More flexible to add parameters to the URL
  • http://www.google.com?q=Python searches for pages related to Python
  • "?" separates the parameters from the rest of the URL
  • If there are multiple parameters, they are separated from each other by "&"
    • http://www.google.ca/search?q=Python&client=firefox
  • Each parameter takes the form "name=value"
  • Spaces are replaced by "+" (so “web programming” becomes “web+programming”)
    • We'll see what to do about actual "+"'s in a moment
• Comment on this slide

Special Characters

• Question: what if you want to include "?" or "&" in a parameter?
• Answer: you URL encode it using "%" followed by its 2-digit hexadecimal code

  • Character	Encoding
    "#"	%23
    "$"	%24
    "%"	%25
    "&"	%26
    "+"	%2B
    ","	%2C
    "/"	%2F
    ":"	%3A
    ";"	%3B
    "="	%3D
    "?"	%3F
    "@"	%40

    Table 20.1: URL Encoding

• Example: to search Google for “grade = A+”, use http://www.google.ca/search?q=grade+%3D+A%2B
• Helper functions:
  • urllib.quote(str) replaces special characters in str with escape sequences
  • urllib.unquote(str) replaces escape sequences with characters
  • urllib.urlencode(params) takes a dictionary, or a list of pairs, and constructs the entire query parameter string

    • import urllib
      
      print urllib.urlencode({'surname' : 'hopper', 'forename' : 'grace'})
      print urllib.urlencode([('surname', 'turing'), ('forename', 'alan')])
      

      surname=hopper&forename=grace
      surname=turing&forename=alan
      

• Comment on this slide

Web Services

• Suppose you want to write a script that actually does search Google
  • Construct a URL: easy
  • Send it and read the response: no problem
  • Parse the response: hmm, there's a lot of junk on the page…
• Many first-generation web applications relied on screen scraping
  • Problem is that whenever the site changes its layout, your application breaks
• Modern web services are designed more intelligently
  • When a client sends a request, it indicates that it wants machine-readable XML, rather than human-readable HTML
  • Much easier to parse
  • Much less likely to change over time
  • An example of a remote procedure call

  • Figure 20.4: Remote Procedure Call Using HTTP

  • Actually, all HTTP requests are remote procedure calls
    • Implicitly invoking a function called “fetch me this file”
• Most web services use the Simple Object Access Protocol (SOAP)
  • Despite its name, it's anything but simple
  • Luckily, there are libraries to hide the details for most widely-used web services
• Example: fetch information about a book from Amazon
  • Amazon has defined an API for web services
  • You need to get a license key in order to use it
    • They're free
    • Allows Amazon to throttle requests to one per second per client
  • Use functions in amazon.py to search by various criteria
  • Result is a list of objects with information about items that matched the search criteria

  • import sys, amazon
    
    # Format multiple authors' names nicely.
    def prettyName(arg):
        if type(arg) in (list, tuple):
            arg = ', '.join(arg[:-1]) + ' and ' + arg[-1]
        return arg
    
    if __name__ == '__main__':
    
        # Get information.
        key, asin = sys.argv[1], sys.argv[2]
        amazon.setLicense(key)
        items = amazon.searchByASIN(asin)
    
        # Handle errors.
        if not items:
            print 'Nothing found for', asin
        if len(items) > 1:
            print len(items), 'items found for', asin
    
        # Display information.
        item = items[0]
        productName = item.ProductName
        ourPrice = item.OurPrice
        authors = prettyName(item.Authors.Author)
        print '%s: %s (%s)' % (authors, productName, ourPrice)
    

  • Note: much more code devoted to creating human-readable output than to getting the information
• Comment on this slide

Summary

• Most computers now spend more time communicating than they do calculating
• Every few years, we put another layer on top of the pile of protocols to make communication easier
  • TCP, HTTP, web services…
• Getting information from the web is now (almost) as easy as getting it from a file
• See in the next lecture how to provide information to others
• Comment on this slide