The MEMS Exchange Application Tools

IPC10
February 6, 2002

A.M. Kuchling
www.amk.ca
amk @ amk.ca

What are MEMS?

MEMS = MicroElectroMechanical Systems
They combine perception, computation, and actuation.

Sensors perceive environment
Electronics process sensory information and make decisions
Actuators control the environment

MEMS Processing

MEMS usually start with a silicon wafer (but not always)
MEMS processes combine IC processes with highly specialized micromachining processes

IC Processes
Oxidation
Diffusion
LPCVD
Photolithography
Epitaxy
Sputtering
Ion Implantation
Etc.

Micromachining Processes
Bulk Micromachining
Surface Micromachining
Wafer Bonding
LIGA
Deep Silicon Reactive Ion Etching
Micromolding
Etc.

What is the MEMS Exchange?

The MEMS Exchange will perform a process sequence in order to create a device. The process sequence can be spread across several individual fabrication facilities for the sake of process and design freedom.

The MEMS Exchange virtual fabrication network allows flexibility in:

discrete processing steps
ordering of processing steps
materials
processing site(s)
volume and cost
response time

Process Catalog

Run Builder

A Process Run

First step toward building Web-based tools.

Checks a sequence for violations of several different rules.

Also graphically displays the layers being built up.

ZODB Overview

An object persistence system for Python
Part of Zope's infrastructure
I initially packaged it for standalone release; a Zope Corp. release is now available.

Features:

Supports transactions
Not fully ACID, though (no Consistency)
Supports different physical storages

ZODB: Example

from ZODB import DB
from ZODB.FileStorage import FileStorage

storage = FileStorage('/tmp/test-filestorage.fs')
db = DB(storage)
conn = db.open()
root = conn.root()
user_db = root['user_db']

newuser = User('amk') 
user_db.users['amk'] = newuser
get_transaction().commit()

Replace FileStorage with BerkeleyStorage to use a different low-level storage mechanism.

ZEO consists of a ClientStorage class which retrieves objects over a socket, and a server that the ClientStorage can talk to.

Making a Class Persistent

Subclass Persistence.Persistent

That's it! There are a few rules to keep in mind, though...

import ZODB
from Persistence import Persistent

class User(Persistent):
    def __init__ (self):
        self.email = None
        self.names = []
    
    def add_name (self, name):
        self.names.append(name)
        self._p_changed = 1

Rules for Persistent Classes

When modifying mutable objects, set dirty bit (_p_changed)
isinstance(), issubclass() don't work.
Some special methods don't work, notably __r*__
__setattr__, __delattr__ must set dirty bit manually

Eventually, an updated version of ZODB for Python 2.2 should make #2 and #3 go away.

MX Database Structure

ZODB Root Objects
BusinessDatabase UserDatabase RunDatabase ...

UserDatabase
BTree: abenard akuchlin gward mhuff nascheme ...

User object

<User at 0000000ef3: akuchlin>
  prefix/first/last/suffix: Mr/Andrew/Kuchling/
  email: 'amk@amk.ca'
  address: <Address at 0000000f2d>
    ...
  _p_changed: false

Address object

<Address at 0000000f2d>
  street1/2/3: 1320 N. Veitch St., #608//
  city/state/zip: Arlington/VA/22201
    ...
  _p_changed: false

Issues

Ad hoc changes (this won't take a minute...)
Typos (self.templtae instead of .template)
Type correctness (this attribute should be a string)
Correctness (ensuring singletons remain singletons)

Solutions

opendb (use the Python interpreter to access our database)
Grouch (a type-checking tool)
zodb_census (counts the population of different objects)
zodb_index (following and finding references)

opendb

Provides a comfy interactive prompt.

Variables for the root objects
Shortcuts for common tasks
History using Michael Hudson's PyRepl

opendb transcript

ludwig akuchlin>opendb
root databases available:
  process_lib
  run_db
  user_db
  ...
 
other variables and functions:
  root
  commit() = get_transaction().commit()
  abort()  = get_transaction().abort()

>>> r = run_db.get_run(113)
>>> r.owner
<User at 839e2f8: akuchlin>
>>> r.owner = user_db.get_user('gward')
>>> r.owner
<User at 83a0348: gward>
>>> commit()        # commit the transaction

Grouch: introduction

Grouch is an after-the-fact type-checker that we run nightly.

Class docstrings describe types of each attribute.
Notifies us of:
- incorrect types
- missing or extra attributes
Does not detect bugs immediately.

(See Greg Ward's presentation in the Lightning Talk session tomorrow.)

Grouch: Docstring format

class ProcessRun(MXPersistent):
    """
    Instance attributes:
      run_id : int
        the unique identifier for this process run
      name : string
        the user-supplied name of this run
      owner : User    # notnone
        the individual user who owns this run
      billed : DateTime | boolean
        whether or not we've billed the customer
      review_comments : PersistentList [Comment]
        An always-growing list of comments...
    """

Grouch: Output of a checking run

Type-checking discovered database errors; here they are:
 
run_db.runs[639].object_versions[6].sequence._key_map['S0002'].wafer_ids['W001']
     .description.resistivity.unit.dims:
  expected attribute 'dimension_names' not found
run_db.runs[639].object_versions[6].sequence._key_map['S0002'].wafer_ids['W001']
     .description.resistivity.unit.dims:
  expected attribute 'num_dimensions' not found
run_db.runs[639].object_versions[6].sequence._key_map['S0002'].wafer_ids['W001']
     .description.resistivity.unit.dims:
  expected attribute 'dimension_powers' not found
make: *** [check] Error 1

zodb_census

Loops through the OIDs in the database, and counts up the number of instances for each class.

ute tools>MX_DB=file:/www/var/mxdb.fs python zodb_census.py
opening database...
expecting to see 306153 objects
maximum expected OID: 00000abe43
OID: 00000abe44 (objects seen: 306153)
census completed
total OIDs attempted: 704068
empty slots seen: 397915
actual objects seen: 306153
objects seen by type:
ActiveVersionCollection          963
Address                         1861
AlignmentMark                     51
BaseProcess                     2324
BusinessDatabase                   1
  ...
OOBTree                           15
OOBucket                         296
Parameter                      29585
ParameterList                  17966
ProcessHierarchy                   1
ProcessLibrary                     1
ProcessModule                    840
ProcessRun                      1885
ProcessSequence                 1885
ProcessStep                    16990
RunDatabase                        1
  ...

zodb_index: Explanation

Builds an index of the references in the ZODB's object graph, and lets you explore it.

-c: builds a fast index of the references in a ZODB.
Time-consuming to build index (10-15 minutes for our DB)
-f: report all OIDs referenced by an object
-r: report all OIDs that refer to an object
-C: report all instances of a given class

Debugging with zodb_index (I)

(Finding a bug, as demonstrated by Greg Ward on our internal mailing list)

First, a mystery:

 
  $ opendb
  [...]
  >>> len(user_db.users)
  775
 
  $ make census
  [...]
  User                             873

There are almost 100 more User objects in the ZODB than users in our UserDatabase object.

Debugging with zodb_index (II)

Let's list those 874 objects, sorted by user ID:

 
  $ ./tools/zodb_index.py -C mems.access.user.User | sort -f -b +4
 
[...]
  00000380d0: <User at 83805c8: akuchlin>
  00000010ca: <User at 83278d8: gward>
  0000000011: <User at 816cd18: Hexon>
  000001a5d1: <User at 8332608: Hexon>
  000001e4b1: <User at 833a138: Hexon>
  0000021943: <User at 83437c0: Hexon>
  0000023c4c: <User at 8346be8: Hexon>
  00000317af: <User at 82c0578: Hexon>
  00000010c9: <User at 83258d0: wbenard>

6 objects for the user "Hexon"?!?

Debugging with zodb_index (III)

Running zodb_index.py -r 0000000011 shows 251 refs to OID 11, so we'll guess that OID 11 is The One True User Object for Hexon, and the others are impostors.

Let's dig deeper...

 
  $ ./tools/zodb_index.py -r 1a5d1 1e4b1 21943 23c4c 317af
  000001a5d1: <User at 81e4768: Hexon>
    000001a5d0: <Review at 8220698>
    000001a5d2: <FabProvider at 82248d0: UMich>
    000001a5d2: <FabProvider at 82248d0: UMich>
    000001a608: <Comment at 8231b88: Hexon>
    0000060994: <Review at 8230a28>
    0000060b00: <Comment at 822ff60: Hexon>
  000001e4b1: <User at 8224550: Hexon>
    000001e4b0: <Review at 8232308>
    000001e4b2: <FabProvider at 822ff00: UMich>
    000001e4b2: <FabProvider at 822ff00: UMich>
    000001e5e7: <Comment at 8233138: Hexon>
  ...

ZODB Enhancements We'd Like to See

Use Python 2.2 features instead of ExtensionClass
(on the way...)
Follow Unix daemon conventions with ZEO
Concurrent access without ZEO?
Getting too big to iterate over the whole database?
(Automatic zodb_index-style indexing?)

We may contribute some of our tools (opendb, MXBase?).

Unit Testing: Sancho

We have over 4000 tests for our basic objects. The existence of this test suite gives us confidence when we refactor the code.

Consider this simple function:

def f(s, val):
    if val < 0:
        raise ValueError, 'val cannot be negative'
    elif val == 42:
        print 'The answer!'
 
    return s * val

Unit Testing Example

from sancho.unittest import TestScenario, \
           parse_args, run_scenarios
import module
 
tested_modules = ['module']
class MyFunctionTest (TestScenario):
    def setup(self): pass
    def shutdown(self): pass
    def check_func(self):
        "Test the function's output: 6"
 
        # Test the null case (val == 0)
        self.test_val( "module.f('', 0)", '')
        self.test_val( "module.f('abc', 0)", '')
 
        # Test the identity (val == 1)
        self.test_val( "module.f('', 1)", '')
        self.test_val( "module.f('abc', 1)", 'abc')
 
        # Test a real case (val == 3)
        self.test_val( "module.f('', 3)", '')
        self.test_val( "module.f('abc', 3)", 'abcabcabc')
 
if __name__ == "__main__":
    (scenarios, options) = parse_args()
    run_scenarios (scenarios, options)

Quixote

Quixote is our environment for building Web applications.

Design goals:

To make the environment as similar to regular Python as possible.
To focus on Web applications that are heavy on programming, light on graphic design.
To avoid magical behaviour.

Features:

Supports regular CGI, FastCGI, and our own mod_scgi.
Uses Python Template Language for generating HTML.
Includes components for form processing, session tracking.

Structure of a Quixote Application

An application is just a Python package whose name is specified in a config file. (mems.ui, quixote.demo, webapp)

webapp/
    __init__.py                     
    module1.py
    module2.py
    pages1.ptl
    pages2.ptl

A URL is mapped to a callable Python object by traversing objects starting from the base package.

http://www/	calls	webapp._q_index
http://www/simple	calls	webapp.simple
http://www/run/	calls	webapp.run._q_index

Example Code

__init__.py:

_q_exports = ["simple", "error", "widgets"]

import sys
from quixote.demo.pages import _q_index
from quixote.demo.widgets import widgets
from quixote.demo.integer_ui import IntegerUI

def simple (request):
    # This function returns a plain text document, not HTML.
    request.response.set_content_type("text/plain")
    return "This is the Python function 'quixote.demo.simple'.\n"

request wraps up:

form input
environment variables,
the response's content type and headers.

It's derived from Zope's HTTPRequest/HTTPResponse classes.

HTML Templating

Most templating syntaxes look like HTML with a bit of additional syntax.

PHP	<? for ($i=1; $i<10; $i++) print $i;>
ASP	<% addr = Request.form("email"); %>
DTML	`<dtml-var "row">` or `<!--#var "row"-->`

Our syntax looks completely different, though still familiar...

How PTL Works

When Python evaluates a lone expression, it discards the result:

def f():
    string.lower('abc')
    return 1

In PTL, the result is converted to a string and appended to the output.

template numbers(n):
    for i in range(n):
        i
        " " # add whitespace

An import hook lets us import PTL files as if they're regular Python modules:

from quixote import enable_ptl
enable_ptl()
from webapp.pages import numbers

PTL Example

pages.ptl:


template _q_index(request):
    print "debug message from the index page"
    """
    <html>
    <head><title>Quixote Demo</title></head>
    <body>
    <h1>Hello, world!</h1>
    """

    "<table>"
    for i in range(10):
        make_row(i)
    "</table>"

template make_row(num):
    "<tr><td colspan=%i>%i</td></tr>" % (num, num)

Why We Like PTL

Flexibility: need to import an arbitrary Python module? Want to write a little helper function? Go ahead...
Readability.
- Indentation makes the structure clear.
- The escape sequence is short, which matters in the face of standard programming practice (factoring out common code).
We already have vi and Emacs modes for it.

Quixote's Special Methods

_q_index (request) : string
When traversal stops at a module or package, this is the default function name that's tried.

_q_access (request)
When found along the way, this function is called and must not raise an exception to let traversal continue.

_q_getname (request, component) : object
When a name isn't found, _q_getname will be called. If it returns an object, traversal will continue with this object.

_q_access(request)

Access control function; traversal can go no further if it raises an exception.

  def _q_access (request):
    from mems.ui.lib.errors import NotLoggedInError
    if request.session.user is None:
        raise NotLoggedInError, \
           ("You must be signed in to view runcards.")

This saves us from having to write checking code for every public function. Instead, we can just put an access restriction on the whole module.

_q_getname(request, component)

"/run/200/" is a more readable URL than "/run/?run_id=200".

_q_getname gets called with the current request and the component of the URL path. If it returns an object, traversal continues with that object.

_q_getname example

mems/ui/run/__init__.py:

def _q_getname (request, component):
    return RunUI(request, component)

class RunUI:
    _q_exports = ['details', 'check', ...]

    def __init__ (self, request, component):
        run_db = get_run_database()
        self.run = run_db.get_run(int(component)) 

    template _q_index (self, request):
        # /run/200/
        ... return index page ...

    template details (self, request):
        # /run/200/details/
        ... return a more detailed page ...

Form Framework

An optional part of Quixote that lives in the quixote.form subpackage, the framework lets us quickly implement basic (and not so basic) HTML forms.

Features:

Contains many different widgets (StringWidget, PasswordWidget, CheckboxWidget, etc.)
It's also possible to write custom widgets (PhysicalValueWidget, MaterialWidget)
Can use hidden form tokens to prevent cross-site request forgery attacks.

Availability

Quixote, Sancho, Grouch:
http://www.mems-exchange.org/software/

Greg Ward will be talking about Grouch tomorrow.

More on the MX toolset:
http://www.amk.ca/python/writing/mx-architecture/

Join the quixote-users mailing list:
http://www.mems-exchange.org/mailman/listinfo/quixote-users/

These slides:
http://www.amk.ca/talks/

MX Application Tools

IPC10 -- February 6, 2002

The MEMS Exchange Application Tools

What are MEMS?

MEMS Processing

What is the MEMS Exchange?

Process Catalog

Run Builder

A Process Run

ZODB Overview

ZODB: Example

Making a Class Persistent

Rules for Persistent Classes

MX Database Structure

Issues

Solutions

opendb

opendb transcript

Grouch: introduction

Grouch: Docstring format

Grouch: Output of a checking run

zodb_census

zodb_index: Explanation

Debugging with zodb_index (I)

Debugging with zodb_index (II)

Debugging with zodb_index (III)

ZODB Enhancements We'd Like to See

Unit Testing: Sancho

Unit Testing Example

Quixote

Structure of a Quixote Application

Example Code

HTML Templating

How PTL Works

PTL Example

Why We Like PTL

Quixote's Special Methods

_q_access(request)

_q_getname(request, component)

_q_getname example

Form Framework

Availability