I've been digging through ~kent/src looking to see how the UCSC browser works, so we can write our own API for grabbing data out of it.
Table descriptions from from .as files (autoSql) and are loaded into tableDescriptions.
There's grp table which lists the defined groups.
The map between tables and which group they belong to appears to come from *.ra files. I haven't found a place in the database where they're stored yet.