Coding Style and Reading Code

Introduction

• Good programming style is very hard to teach
  • All the “rules” are either banal or overly restrictive
    • Saying, “Make methods short, but not too short” doesn't really help
    • Neither does saying, “No method shall be longer than sixty lines”
  • Matters aren't helped by the fact that the strength of programmers' opinions is inversely proportional to the amount of data they have
• But good style is tremendously important
  • Well-written code is less likely to contain errors
  • And is much easier to maintain
• Rather than teaching you how to write well, this lecture will focus on how to read code
  • How to figure out how the code works
  • How to find what you need to know before making a change
• Hope that you'll pick up ideas about good style along the way
• For a comprehensive guide to how open source Unix applications are (or should be) structured, try [Spinellis 2003]
• Comment on this slide

Why Read Code?

• People doing creative work in almost every field routinely inspect, dissect, and critique what's come before
  • Would you hire an architect who'd never looked at anyone else's buildings?
• Programming is a sad exception
  • Most students only ever read one- or two-page fragments in textbooks
  • Like reading sonnets, then being asked to write a novel
• Open source has made millions of lines of code available for inspection
  • Some good, some bad, most average
• Knowing how to read code is as useful as knowing how to read a proof
  • Learn by example how to accomplish specific things
  • Figure out how to make changes in a particular program
  • Figure out how to use the program
  • Code inspections are essential for achieving and maintaining quality
• Comment on this slide

Seven Plus or Minus

• Known since the 1950s that human short term memory can hold 7±2 items [Hock 2004]
  • Seven random digits (as in phone numbers)
  • Seven tasks that still have to be done
  • Note: this is just an average
    • Some prodigies can remember thousands of pieces of information for short periods of time
• If we try to remember more than that, we either:
  • Make mistakes, or
  • Create chunks so that we can remember things at a higher level
    • Musicians don't remember individual chords; they remember IV-V-I progression
    • Chess players remember “castled kingside”, rather than the positions of five pieces

    • Figure 15.1: Chunking in Short-Term Memory

  • [Chase & Simon 1973] studied what happened when novice and master chess players are shown actual and random positions
    • Masters (naturally) remember actual positions better
    • But they remember random positions worse
    • They're “seeing” patterns that aren't there

  • Figure 15.2: Actual Chess Position

    Figure 15.3: Retention of Actual Chess Position

  • Figure 15.4: Random Chess Position

    Figure 15.5: Retention of Random Chess Position

• What does this have to do with programming?
  • When you are reading and writing code, you have to keep a bunch of facts straight in your head for a short period of time
    • What is this function supposed to return?
    • What parameters does it have, and what do they mean?
    • What does this loop's index refer to?
    • Etc.
  • The more odds and ends you require readers (including yourself) to keep track of, the greater their error rates will be
    • Fundamental rule of good programming style is therefore to limit the amount of information needed at any one time
  • Note that a hundred one-line methods is not an improvement over one hundred-line method
• A related rule: the greater the difference, the more likely we are to notice it
  • So every difference in naming and layout ought to mean something…
  • …because readers' brains will assume that it does
  • Conversely, every significant difference ought to be visually distinct
• Comment on this slide

What Does This Function Do?

• Start with a simple question: how to figure out what this code does?

  • # Fill in citations.
    def bibRef(root, bibOutput, bibInfo):
        cites = root.getElementsByTagName('cite')
        for cite in cites:
            ref = str(cite.getAttribute('ref'))
            if (not ref) or (ref not in bibInfo):
                print "Cannot satisfy '%s'" % ref
                assert False
            cite.setAttribute('display', bibInfo[ref])
            cite.setAttribute('href', '%s#%s' % (bibOutput, ref))
    

• First question: what's the context?
  • In this case, it's part of a program to fill in citations
  • <cite ref="castro-xml"/> becomes <cite display="Castro 2002" href="bib.html#castro-xml" ref="castro-xml"/>
• Ways to answer the question:
  • Guess based on function name
    • Assume bibRef stands for “bibliographic reference”
    • But it isn't a particularly informative name
  • Read embedded documentation (such as comments)
    • “Fill in citations.” is more helpful than bibRef
    • Still doesn't tell us what root, bibOutput, and bibInfo are
    • Or what side effects (if any) this function has
  • Read the function body
    • For each citation element in body…
    • …get the value of the element's ref attribute…
    • …make sure it's in bibInfo (presumably a dictionary)…
    • …set the citation's display and href attributes
    • Hm…that means bibOutput must be the name of the output file
    • Notice how the explanation creates larger chunks from raws lines of code
  • Consult external documentation
    • In this case, there doesn't appear to be any
  • Examine call(s) to the function
    • First job is to find it (or them)
    • grep bibRef expandrefcite.py finds exactly one call

    •         # Substitute information.
              for filename in args:
                  filename, _, _, root = getFile(filename)
                  genTime(root)
                  crossRef(root, xrefInfo)
                  bibRef(root, bibOutput, bibInfo)
                  outfile = open(filename, 'w')
                  outfile.write(root.toxml('utf-8'))
                  outfile.close()
      

    • Looking higher up, see that bibOutput is initialized to bib.html, so our guess was pretty good
• You use these techniques (consciously or otherwise) whenever you are reading code
  • The purpose of good style is to improve the odds of them working
• A good way to think about size and scope is the notion of a program slice
  • The subset of names in scope at a particular statement that are needed to understand what that statement does
  • If the slice is much smaller than the method itself, the method is probably bloated
• Comment on this slide

Naming

• Names of files, classes, methods, variables, and other things are the most visible clue to purpose
  • A variable called temperature probably doesn't store a user name or the number of pottery shards found at a dig site
• Should therefore choose names that are both meaningful and readable
  • current_surface_temperature_of_probe is meaningful, but not readable
  • cstp is easier to read, but hard to understand
• The smaller the scope of a name, the more compact it can be
  • It's OK to use i and j for indices in tightly-nested for loops
  • But not OK if the loop bodies are several pages long
    • Of course, they shouldn't be anyway…
• The wider the scope of a name, the more descriptive it has to be
  • Call a class ExperimentalRecord, rather than ER or ExpRec
• Compare:

  • Before

    After

    import reader,splitter,transpose import sys,os a=[] b=[] c=[] d=sys.argv[1] a=reader.readLinesFromFile(d) b=splitter.splitIntoSections(a) c=d.split('.') for i in range(len(b)): if os.path.isfile('%s.%d.dat'%(c[0],i+1)): print '%s.%d.dat already exists and cannot be overwritten!'%(c[0],i+1) break else: output=file('%s.%d.dat'%(c[0],i+1),'w') print>>output,transpose.transpose(b[i]) output.close()

    import sys, os import reader, splitter, transpose inputFileName = sys.argv[1] lines = reader.readLinesFromFile(inputFileName) sections = splitter.splitIntoSections(lines) fileNameStem = inputFileName.split('.')[0] for i in range(len(sections)): outputFileName = '%s.%d.dat' % (fileNameStem, i+1) if os.path.isfile(outputFileName): print '%s already exists and cannot be overwritten!' % outputFileName break else: output = file(outputFileName, 'w') print >> output, transpose.transpose(sections[i]) output.close()

• Use naming conventions to indicate purpose
  • Constants written ALL_UPPER_CASE_WITH_UNDERSCORES
  • Class names written using CamelCase with first letter capitalized
    • Name “camel case” comes from the humps in the middle
  • Parameters, methods, local variables, etc., use camelCase with first letter in lower case
• Many other conventions make just as much sense
  • Write parameters, etc., using underscores, as in current_maximum
  • Write class members like this_, _this, or m_this
    • Not as useful in Python, where you have to refer to members as self.this anyway
    • But useful in C++, Java, and other languages
  • Use an agreed list of abbrevations
    • currAveTemp instead of currentAverageTemperature
    • “Agreed” is important: mixing currAvTemp and currAveTemp produces hard-to-spot bugs
• Most important thing is to be consistent
  • Anything consistent is readable after a while
    • Just watch kids learning to read French, Punjabi, and Korean
  • But your brain automatically notices visual differences, and assumes they're important
  • It actually does take longer to read code written in a mix of styles…
  • …and the error rate is higher
• Conform to PEP-008: Python Style Guide unless you have a really good reason not to
• Comment on this slide

Idioms

• Every language (human or computer) has idioms
  • Sayings or phrases whose meaning does not follow from the meaning of the words it is made up of
    • E.g., “face the music” does not mean “look at the orchestra”
  • A way to aggregate raw data into chunks
• As you read programs, look for uses of common idioms
  • Example: C/C++ and Java allow assignment in the middle of an expression
    • z = (y = x) + 2 assigns the value of x to y, then adds 2 to that and assigns it to z
  • No one would (or should) ever use it in the manner show above, but it makes certain kinds of loops more compact

    • Without Idiom	Using Idiom
      line = infile.readLine(); while (line != null) { ...do something with line... line = infile.readLine(); }	while ((line = infile.readLine()) != null) { ...do something with line... }

    • Assign next line (if any) to text, then check for end of input, all in the top of the loop
  • This can be even more compact in C and C++, which treat null as equivalent to false

    • while (line = infile.readLine()) {
          ...do something with line...
      }
      

  • But Python doesn't allow assignment in the middle of expressions
    • And in older versions, you couldn't iterate directly over a file using for line in input:
    • Solution was an idiom that looked like an infinite loop, but had a break statement in it to handle end-of-input

    • while 1:
          line = infile.readline()
          if not line:
              break
          ...do something with line...
      

    • Note: using 1 instead of True, since older Pythons didn't define True either
  • You can figure out what the code is doing even if you don't know the idiom…
    • …but knowing the idiom makes it a lot simpler for your brain to chunk what it's reading
• Style guideline: use your language's idioms
  • If everyone else writes their code a certain way, write yours that way too
    • I never liked Python's while 1: break idiom, but I used it
  • Two ways to learn idioms:
    • Books on programming style (e.g., the “Effective” series from Addison-Wesley)
    • Reading other people's code
• Note: idioms evolve over time
  • Languages gain new features, programmers learn new tricks, etc.
  • People who have grown up with Java tend to write cleaner C than people who learned it as a first language
  • So when reading code, prefer the style of newer code over that of older stuff
• Comment on this slide

Style Tools

• A growing number of tools exist to check and enforce style guidelines
  • Look for things that the language spec and compilers allow, but which make code harder to read
• PyLint
  • Parses programs to create an annotated syntax tree
    • A data structure that represents classes, loops, variables, etc.

    • Figure 15.6: Annotated Syntax Tree

  • Then searches the tree to find things that match problem patterns
    • Each pattern implemented as a class that looks for certain things in the tree
    • Which means that users can write new checkers if they want to

  • class Metal:
        name = "aluminum"
    
        def __init__(self):
            self.name = arg
    
        def meth(self, val):
          self.v = val
    

  • pylint badcode.py
    

    ************* Module badcode
    W:  8: Bad indentation. Found 6 spaces, expected 8
    W:  0: Missing docstring
    W:  0: Missing required attribute "__revision__"
    W:  1:Metal: Missing docstring
    E:  5:Metal.__init__: Undefined variable 'arg'
    W:  7:Metal.meth: Missing docstring
    R:  1:Metal: Not enough public methods (1/2)
    W:  8:Metal.meth: Attribute 'v' defined outside __init__
    

• PyChecker
  • Imports the module (or modules)
    • If the code doesn't run, PyChecker can't analyze it
  • Uses compiled byte codes as well as source when doing analysis

  • pychecker badcode.py
    

    Warnings...
    
    badcode.py:5: No global (arg) found
    

• Similar tools exist for [name of your language goes here]
  • Most check for much more than just indentation and naming conventions
  • Can also find:
    • Dead code (i.e., variables and methods that are never used)
    • Unhandled exceptions
    • Duplicate code
    • And much more
• Use them during development
  • Don't check anything into version control unless it passes a style check
  • Take their warnings very seriously when debugging
    • If someone was sloppy enough to make style mistakes, they probably made more serious mistakes as well
• Use them when reading code
  • Many tools can reformat code so that it obeys style guidelines
    • Sometimes called pretty-printers
  • Do not reformat code just because you can
    • Makes it difficult to track and merge changes in version control
  • But reformatting a copy to make it easier to read can save you headaches
• Comment on this slide

What About Documentation?

• In an ideal world, all code would be accurately documented
  • And the Beatles would still be performing as a group…
• In this world, documentation is often missing
  • Or worse, it may have been up to date three years ago, but no longer describes the software correctly
• Many kinds of documentation are useful
  • Embedded in code to explain tricky bits
    • “This is a while loop, instead of a for, because it may delete items from the list as it goes”
  • Associated with class/method headers to explain what they're for
    • Documentation for a library
    • “This function returns the first pair of non-overlapping subsequences that match the input pattern, or null otherwise”
  • Architectural descriptions
    • The architecture of a program is what you draw on the whiteboard when explaining its internals to other people
    • May show relationships between classes, major modules, data flow, etc.
    • Often the most important documentation of all, since it's what gives other programmers a mental map of how everything fits together
  • Requirements
    • Description of what needs the software is supposed to meet
    • “If more than 100 events arrive in a second, the seismograph interface must store them in a queue“ tells you to look for a seismograph interface, and a queue that feeds it data
  • User guide
    • As above, knowing what the software is supposed to do often helps figure out how it works
• It's increasingly fashionable to embed some documentation in code, then extract and format it
  • Javadoc (for Java) translates specially-formatted comments into HTML

    • Java

      HTML

      /** * Returns the least common ancestor of two species based on DNA * comparison, with certainty no less than the specified threshold. * Note that getConcestor(X, X, t) returns X for any threshold. * * @param left one of the base species for the search * @param right the other base species for the search * @param threshold the degree of certainty required * @return the common ancestor, or null if none is found * @see Species */ public Species getConcestor(Species left, Species right, float threshold) { ...implementation... }

      <p><strong>getConcestor</strong></p> <p><code>public&nbsp;Species&nbsp;getConcestor(Species&nbsp;left,&nbsp;Species&nbsp;right,&nbsp;float&nbsp;threshold)</code></p> <blockquote> <p>Returns the least common ancestor of two species based on DNA comparison, with certainty no less than the specified threshold. Note that getConcestor(X, X, t) returns X for any threshold.</p> <p><strong>Parameters:</strong></p> <blockquote> <p><code>left</code> - one of the base species for the search <p><code>right</code> - the other base species for the search <p><code>threshold</code> - the degree of certainty required </blockquote> <p><strong>Parameters:</strong></p> <blockquote> <p>the common ancestor, or null if none is found</p> </blockquote> <p><strong>See Also:</strong></p> <blockquote> <p><code>Image</code></p> </blockquote> </blockquote>

  • Python's PyDoc module does similar things
• Embedded documentation is more likely to be up to date than external documentation
  • It's right in front of programmers as they make changes
  • Although that's no guarantee…
• Comment on this slide

Traceability

• Reproducibility is one of the things that makes science science
  • Anyone can (at least in principle) repeat your experiment to verify your results
• Traceability is a key part of this in computational science
  • What program(s) did you use?
  • Where did your data come from, and what did you do to it?
• Always include version control information in source files
  • Subversion can be told to look for strings like $Revision: 421$ when you submit changes, and replace the text after the colon with the appropriate information
  • Allows the file to carry its identity with it when it's printed, archived, or emailed
  • Usually put the keyword inside a string, so that version information is available inside the program

  • __version__ = "$Revision: 423 $"
    
    if __name__ == '__main__':
        print __version__
    

• Data files are source files too!
  • Every original data file should go under version control, and have revision information stored in it
    • Can record author and date of last change as well as version number
  • Carry this information through when processing files
    • E.g., input file biomes.dat has a header $Revision: 421$
    • Is processed by version 37 of ecoanalyzer.py
    • So include the following header (as comments) in the output file:

    • # $Revision: $
      # From: biomes.dat version 421
      # By: ecoanalyzer.py version 37
      # On: 2005-09-22 12:14:07 EST
      

• Comment on this slide

Executable Documentation

• Also increasingly fashionable to talk about executable documentation
  • Things that describe the program in ways that the computer can check
• Assertions
  • Assertions at the top of a method are usually checking pre-conditions that must be true for the method to run properly
  • Those at the bottom are guarantees of what will be true after the method has run
    • Unless there are return statements embedded in the middle of the method…
    • …which is another reason they're a bad idea
  • Assertions in the middle of methods are usually checking:
    • Invariants (i.e., number of items in the searched list, plus the number in the toSearch list, stays constant)
    • Old bugs: common to introduce assertions while debugging, then leave them in to detect the bug's reappearance
  • When reading code, assertions let you compare your current understanding of the code with what it actually does
    • If you see assert x < y, and you don't know why x has to be less than y at that point, you've lost track
  • So when writing code, add assertions:
    • To detect errors and mis-use early
    • To help the next person understand how the code is supposed to behave
• Tests
  • Each test is really just a super-assertion
    • Each test that's up to date and runs correctly, that is
  • They are not a substitute for documentation
    • Which is easier to read, 2³ test cases, or one sentence explaining the relationship between a method's three Boolean input parameters?
  • But they are a definitive answer in cases of ambiguity
  • When writing, use test-driven development to ensure that tests exist, and are kept up to date as code changes
  • When reading, look for tests that use the classes or functions you're interested in, and make sure you understand why the given input produces the checked-for output
• Build
  • Every medium to large project has infrastructure to compile source files, create installers, run tests, etc.
    • Make, Ant, SCons, etc.
  • Knowing how the software is built can tell you a lot about what to look for
    • Entire directories of source code may be ignored depending on the operating system you're on
    • Different modules may be used depending on whether it's the Lite or Professional version
  • Build control files are notoriously difficult to read
    • Often easier to perform a build, and look at its output to see what files are involved, what flags are used, etc.
  • When writing code, try to keep the build as simple as possible
    • If any parts of the software are only conditionally included, make the conditions very prominent
    • If you're linking non-standard libraries, make sure they're flagged too
• Comment on this slide

Active Reading

• Sometimes, the only way to understand a piece of code is to get your hands dirty
  • Logging
  • Running in a debugger
  • Dynamic code inspection tools
• Most languages have logging libraries that let programmers control debugging output
  • Typical call is log("GRID", LOG_DEBUG, "Refining a %d X %d grid", dimX, dimY)
  • First argument specifies topic (sometimes called “channel”)
    • Allows readers to filter messages (e.g., select only those related to grid updates)
  • Second specifies level: only print message if user has asked for that much detail

    • Level	Meaning
      DEBUG	Debugging output (of interest to programmers only)
      INFO	Normal operations (e.g., a user logging in)
      WARN	Something that requires an administrator's attention (e.g., a user is out of disk space)
      ERROR	Something wrong with the program that it can recover from (e.g., unable to open its own configuration file)
      CRITICAL	System about to crash, missiles about to launch, yet another bad 1970s TV show being turned into a movie

      Table 15.1: Typical Logging Levels

  • At start of program, specify where to send logging output
    • To screen
    • To a file
    • To a file that “rolls over” when it reaches a certain size
    • To a database
    • Through a socket to a collector on some other machine
    • Etc.
  • When writing code, add logging statements to record important events
    • Set to “no output” by default to avoid annoying users
    • When program starts to behave strangely, have users turn on logging and send you the trace
  • Encourages cleaner coding
    • If there's no point at which you can print, “Rotation complete”, it'll be hard for readers to figure out which code is responsible for doing rotations
• Debuggers aren't just for finding bugs
  • Stepping through code is often the easiest way to figure out how it works
    • First pass: step over functions, rather than into them
    • Then re-run, and dig down to the next level of detail
  • Hardest part is often figuring out where to start
    • Depends on the architecture
    • Command-line applications: start with main (or, in Python, __name__ == '__main__')
    • If the code plugs into a larger framework, set breakpoints on key methods, then examine the call stack to see how you got there
  • Some debuggers are more graphical than others
    • Data Display Debugger (DDD) can draw box-and-arrow diagrams of data structures

    • Figure 15.7: Data Display Debugger (Courtesy of Andreas Zeller, The DDD Manual)

• Integrated Development Environments
  • A good IDE is more than just a fancy editor and a symbolic debugger
  • Good ones (like Eclipse) also include code browsers
    • Keep track of where classes, methods, and variables are defined
    • Single click to jump from a variable to its definition, then to the definition of its class
    • Another click to search for variables of a certain class (or its subclasses)
    • And much, much more
  • Similar tools available for old-fashioned text editors
    • Ctags parses source files and produces a database of the definitions they contain
    • Single editor command jumps from use to definition
    • Currently works for over 30 languages
• Comment on this slide

Summary

• The most efficient way to read code is almost always to:
  • Print it out
    • Paper has 4-10 times the resolution of even the best screen
  • Sit somewhere comfortable
    • Away from email, the web, your pager, and other distractions
  • Find an entry point (like main)
  • Trace execution from there, skipping over stuff that looks straightforward
    • E.g., argument parsing and file I/O
  • Put a question mark beside everything that doesn't immediately make sense
  • Once you're done, go back and try to answer your questions
• This also happens to be a good way to do a code review
  • Read a colleague's code in order to:
    • Find bugs
    • Make sure she is following style guidelines
    • Make sure she implemented what you asked for
  • Code reviews are a more effective way to find bugs than running ad hoc tests
    • Which is yet another good reason to learn how to read code
• Comment on this slide