The MEMS Exchange Application Architecture

DC Zope/Python User Group
January 16, 2002

A.M. Kuchling
www.amk.ca
amk @ amk.ca

Topics

What are MEMS?

MEMS = MicroElectroMechanical Systems
They combine perception, computation, and actuation.

Diversity of MEMS Applications

Inertial Measurement Accelerometers, gyroscopes, vibration sensors
Microfluidics Gene chips, lab on a chip, chemical sensors, flow controllers, microvalves, micronozzles
Optical MEMS Optical switches, displays, adative optics
Pressure Measurement Auto sensors, medical sensors, industrial sensors
RF Technology RF switches, filters, variable capacitors, inductors, antennas, phase shifters, scanned aperatures
Other Data storage, picosatellites, actuators, etc.

MEMS Builds on Microelectronics Manufacturing

MEMS Processing

IC Processes
Oxidation
Diffusion
LPCVD
Photolithography
Epitaxy
Sputtering
Ion Implantation
Etc.
 Micromachining Processes
Bulk Micromachining
Surface Micromachining
Wafer Bonding
LIGA
Deep Silicon Reactive Ion Etching
Micromolding
Etc.

Some facts about MEMS

Some facts about MEMS (cont'd)

What is the MEMS Exchange?

The MEMS Exchange will perform a process sequence in order to create a device. The process sequence can be spread across several individual fabrication facilities for the sake of process and design freedom.

The MEMS Exchange virtual fabrication network allows flexibility in:

Current Fabs

Academic

Commercial:

Using the MEMS Exchange

Process Catalog

A Process Run

Run Builder

MX User's Personal Page

Process Rule Checker

Entering Metrology

First step toward building Web-based tools.

Checks a sequence for violations of several different rules.

Also graphically displays the layers being built up.

ZODB Primary Objects

ProcessLibrary  <<database root
  BaseProcess
    Parameter
      PhysicalValue
  ProcessModule
    ...

UserDatabase    <<database root
  User
  Group

RunDatabase     <<database root
  ProcessRun
    ProcessSequence
      [ProcessStep1, ...]         
         Parameter

TemplateLibrary, BusinessDatabase, MaterialDatabase, ...

Why Not Use an RDBMS?

Instead, we use the ZODB. Our database has:

ZODB Overview

Features:

Note: ACID = Atomicity, Consistency, Isolation, Durability

ZODB: Example

from ZODB import DB
from ZODB.FileStorage import FileStorage

storage = FileStorage('/tmp/test-filestorage.fs')
db = DB(storage)
conn = db.open()
root = db.root()
user_db = root['user_db']

newuser = User('amk') 
userdb.users['amk'] = newuser
get_transaction().commit()

Replace FileStorage with BerkeleyStorage to use a different low-level storage mechanism.

ZEO consists of a ClientStorage class which retrieves objects over a socket, and a server that the ClientStorage can talk to.

Making a Class Persistent

That's it! There are a few rules to keep in mind, though...

import ZODB
from Persistence import Persistent

class User(Persistent):
    def __init__ (self):
        self.email = None
        self.names = []
    
    def add_name (self, name):
        self.names.append(name)
        self._p_changed = 1

Rules for Persistent Classes

  1. When modifying mutable objects, set dirty bit (_p_changed)
  2. isinstance(), issubclass() don't work.
  3. Some special methods don't work either; __cmp__, __r*__, ...)
  4. __setattr__, __delattr__ must set dirty bit manually

An updated version for Python 2.2 should make #2 and #3 go away (eventually).

MX Database Structure

ZODB Root Objects
BusinessDatabase UserDatabase RunDatabase ResultsDatabase
UserDatabase
BTree: abenard akuchlin gward mhuff nascheme
User object 
<User at 0000000000000ef3: akuchlin>
  prefix/first/last/suffix: Mr/Andrew/Kuchling/
  email: 'amk@amk.ca'
  address: <Address at 0000000000000f2d>
    ...
  _p_changed: false
Address object 
<Address at 0000000000000f2d>
  street1/2/3: 1320 N. Veitch St., #608//
  city/state/zip: Arlington/VA/22201
    ...
  _p_changed: false

Issues

Solutions

expire_sessions

opendb

Provides a comfy interactive prompt.

opendb transcript

ludwig akuchlin>opendb
root databases available:
  template_lib
  process_lib
  run_db
  user_db
  business_db
  results_db
  session_manager
  shared_cache
 
other variables and functions:
  database
  connection
  root
  commit() = get_transaction().commit()
  abort()  = get_transaction().abort()
  sync()   = connection.sync()

>>> r = run_db.get_run(113)
>>> r.owner
<User at 839e2f8: akuchlin>
>>> r.owner = user_db.get_user('gward')
>>> r.owner
<User at 83a0348: gward>
>>> commit()        # commit the transaction

Grouch: introduction

Grouch is an after-the-fact type-checker that we run nightly.

Grouch: Docstring format

class ProcessRun(MXPersistent):
    """
    Instance attributes:
      run_id : int
        the unique identifier for this process run
      name : string
        the user-supplied name of this run
      owner : User    # notnone
        the individual user who owns this run
      billed : DateTime | boolean
        whether or not we've sent a bill to the customer for this run
      review_comments : PersistentList [Comment]
        An always-growing list of comments...
    """

Grouch: Output of a checking run

Type-checking discovered database errors; here they are:
 
run_db.runs[639].object_versions[6].sequence._key_map['S0002'].wafer_ids['W001']
     .description.resistivity.unit.dims:
  expected attribute 'dimension_names' not found
run_db.runs[639].object_versions[6].sequence._key_map['S0002'].wafer_ids['W001']
     .description.resistivity.unit.dims:
  expected attribute 'num_dimensions' not found
run_db.runs[639].object_versions[6].sequence._key_map['S0002'].wafer_ids['W001']
     .description.resistivity.unit.dims:
  expected attribute 'dimension_powers' not found
make: *** [check] Error 1

zodb_census

Loops through the OIDs in the database, and counts up the number of instances for each class. 
ute tools>MX_DB=file:/www/var/mxdb.fs python zodb_census.py
opening database...
expecting to see 306153 objects
maximum expected OID: 00000000000abe43
OID: 00000000000abe44 (objects seen: 306153)
census completed
total OIDs attempted: 704068
empty slots seen: 397915
actual objects seen: 306153
objects seen by type:
ActiveVersionCollection          963
Address                         1861
AlignmentMark                     51
BaseProcess                     2324
BusinessDatabase                   1
  ...
OOBTree                           15
OOBucket                         296
Parameter                      29585
ParameterList                  17966
ProcessHierarchy                   1
ProcessLibrary                     1
ProcessModule                    840
ProcessRun                      1885
ProcessSequence                 1885
ProcessStep                    16990
RunDatabase                        1
  ...

zodb_index: Explanation

Builds an index of the references in the ZODB's object graph, and lets you explore it.

Debugging with zodb_index (I)

(Finding a bug, as demonstrated by Greg Ward on our internal mailing list)

First, a mystery:

 
  $ opendb
  [...]
  >>> len(user_db.users)
  775
 
  $ make census
  [...]
  User                             873

There are almost 100 more User objects in the ZODB than users in our UserDatabase object.

Debugging with zodb_index (II)

Let's list those 874 objects, sorted by user ID:

 
  $ ./tools/zodb_index.py -C mems.access.user.User | sort -f -b +4
 
[...]
  00000000000380d0: <User at 83805c8: akuchlin>
  00000000000010ca: <User at 83278d8: gward>
  0000000000000011: <User at 816cd18: Hexon>
  000000000001a5d1: <User at 8332608: Hexon>
  000000000001e4b1: <User at 833a138: Hexon>
  0000000000021943: <User at 83437c0: Hexon>
  0000000000023c4c: <User at 8346be8: Hexon>
  00000000000317af: <User at 82c0578: Hexon>
  00000000000010c9: <User at 83258d0: wbenard>

6 objects for the user "Hexon"?!?

Debugging with zodb_index (III)

Running zodb_index.py -r 0000000000000011 shows 251 refs to OID 11, so we'll guess that OID 11 is The One True User Object for Hexon, and the others are impostors.

Let's dig deeper...

 
  $ ./tools/zodb_index.py -r 1a5d1 1e4b1 21943 23c4c 317af
  000000000001a5d1: <User at 81e4768: Hexon>
    000000000001a5d0: <Review at 8220698>
    000000000001a5d2: <FabProvider at 82248d0: UMich>
    000000000001a5d2: <FabProvider at 82248d0: UMich>
    000000000001a608: <Comment at 8231b88: Hexon at 2001-04-27 14:34:31.51>
    0000000000060994: <Review at 8230a28>
    0000000000060b00: <Comment at 822ff60: Hexon at 2001-04-27 14:34:31.51>
  000000000001e4b1: <User at 8224550: Hexon>
    000000000001e4b0: <Review at 8232308>
    000000000001e4b2: <FabProvider at 822ff00: UMich>
    000000000001e4b2: <FabProvider at 822ff00: UMich>
    000000000001e5e7: <Comment at 8233138: Hexon at 2001-04-25 15:58:12.17>
  0000000000021943: <User at 81e3a38: Hexon>
    0000000000021942: <Review at 8232ca0>
  ...

ZODB Enhancements We'd Like to See

Unit Testing

We have over 4000 tests for our basic objects. The existence of this test suite gives us confidence when we refactor the code.

Consider this simple function:

def f(s, val):
    if val < 0:
        raise ValueError, 'val cannot be negative'
    elif val == 42:
        print 'The answer!'
 
    return s * val

Unit Testing Example

from quixote.test.unittest import TestScenario, parse_args, run_scenarios
import module
 
tested_modules = ['module']
class MyFunctionTest (TestScenario):
    def setup(self): pass
    def shutdown(self): pass
    def check_func(self):
        "Test the function's output: 6"
 
        # Test the null case (val == 0)
        self.test_val( "module.f('', 0)", '')
        self.test_val( "module.f('abc', 0)", '')
 
        # Test the identity (val == 1)
        self.test_val( "module.f('', 1)", '')
        self.test_val( "module.f('abc', 1)", 'abc')
 
        # Test a real case (val == 3)
        self.test_val( "module.f('', 3)", '')
        self.test_val( "module.f('abc', 3)", 'abcabcabc')
 
if __name__ == "__main__":
    (scenarios, options) = parse_args()
    run_scenarios (scenarios, options)

Example output

When run, this test case prints: 
ute /tmp>python test.py
FunctionTest:
  ok: Test the function's output ('func') (6 tests passed)
passed) ok: 9 tests passed
ute /tmp>

Or, in case of failure:

ute /tmp>python test.py
MyFunctionTest:
  not ok: Test the function's output ('func') (6 tests expected, 6 run, 2 failed)
    not ok: module.f('', 0) != '' (raised ValueError: "val cannot be negative")
      failed at test.py, line 12 (in 'check_func()'):
        self.test_val( "module.f('', 0)", '')
        File "/home/amk/src/mems/quixote/test/unittest.py", line 337, in test_val
          val = eval (code, globals, locals)
        File "<string>", line 0, in ?
        File "module.py", line 3, in f
          raise ValueError, 'val cannot be negative'
    not ok: module.f('abc', 0) != '' (raised ValueError: "val cannot be negative")
      ...
not ok: 6 tests expected, 6 run, 2 failed
ute /tmp>

Code Coverage

ute /tmp>python test.py -c
MyFunctionTest:
  ok: Test the function's output ('func') (6 tests passed)
passed) ok: 6 tests passed
code coverage:
  module:  83.3% (5/6)
ute /tmp>python test.py -c -v
  ... additional output while running the tests deleted ...
ok: 6 tests passed
code coverage:
  module:
      .
    10: def f(s, val):
      9:    if val < 0:
      1:        raise ValueError, 'val cannot be negative'
      8:    elif val == 42:
  >>>>>>        print 'The answer!'
      .
      8:    return s * val
  83.3% (5/6)
kronos /tmp>

Testing a tree of modules

ute proto3>~/src/mems/tools/run_tests.py -r .
looking for test scripts...found 45
ok: lib/test/test_pvalue.py: 118 tests passed
ok: lib/test/test_range.py: 129 tests passed
ok: lib/test/test_unit.py: 109 tests passed
...
ok: template/test/test_eqtemplate.py: 14 tests passed
ok: prc/test/test_inter.py: 0 tests passed
ok: 4000 tests passed

Code coverage:

  grouch.valuetype: 36.6% (168/459)
  mems.run.process_run: 43.1% (202/469)
  mems.tools.unittest: 45.4% (237/522)
  mems.lib.base: 46.9% (75/160)
  grouch.util: 48.1% (26/54)
  mems.process.process_module: 52.4% (100/191)
  ...

Quixote

Quixote is our environment for building Web applications.

Design goals:

Features:

Structure of a Quixote Application

An app is a Python package whose name is specified in a config. file. (mems.ui, quixote.demo)

webapp/
    __init__.py                     
    module1.py
    module2.py
    pages1.ptl
    pages2.ptl

A URL is mapped to a callable Python object by traversing objects starting from the base package.

http://www/ calls webapp._q_index
http://www/simple calls webapp.simple
http://www/run/ calls webapp.run._q_index

Example Code

__init__.py:

_q_exports = ["simple", "error", "widgets"]

import sys
from quixote.demo.pages import _q_index
from quixote.demo.widgets import widgets
from quixote.demo.integer_ui import IntegerUI

def simple (request):
    # This function returns a plain text document, not HTML.
    request.response.set_content_type("text/plain")
    return "This is the Python function 'quixote.demo.simple'.\n"

request wraps up:

HTML Templating

Most templating syntaxes look like HTML with a bit of additional syntax.

PHP 
<? for ($i=1; $i<10; $i++) print $i;>
ASP 
<% addr = Request.form("email"); %>
DTML <dtml-var "row"> or <!--#var "row"-->

Our syntax looks completely different, though still familiar...

Python Template Language

pages.ptl:


template _q_index(request):
    print "debug message from the index page"
    """
    <html>
    <head><title>Quixote Demo</title></head>
    <body>
    <h1>Hello, world!</h1>
    """

    "<table>"
    for i in range(10):
        make_row(i)
    "</table>"

template make_row(num):
    "<tr><td colspan=%i>%i</td></tr>" % (num, num)

How PTL Works

When Python evaluates a lone expression, it discards the result:

def f():
    string.lower('abc')
    return 1

In PTL, the result is converted to a string and appended to the output.

template numbers(n):
    for i in range(n):
        i
        " " # add whitespace

An import hook lets us import PTL files as if they're regular Python modules:

from quixote import enable_ptl
enable_ptl()
from webapp.pages import simple

Why We Like PTL

Quixote's Special Methods

_q_index (request) : string
When traversal stops at a module or package, this is the default function name that's tried.

_q_access (request)
When found along the way, this function is called and must not raise an exception to let traversal continue.

_q_getname (request, component) : object
When a name isn't found, _q_getname will be called. If it returns an object, traversal will continue with this object.

_q_access(request)

Access control function; traversal can go no further if it raises an exception.

  def _q_access (request):
    from mems.ui.lib.errors import NotLoggedInError
    if request.session.user is None:
        raise NotLoggedInError, ("You must be signed in to view runcards.")

This saves us from having to write checking code for every public function. Instead, we can just put an access restriction on the whole module.

_q_getname(request, component)

"/run/200/" is a more readable URL than "/run/?run_id=200".

_q_getname gets called with the current request and the component of the URL path. If it returns an object, traversal continues with that object.

mems/ui/run/__init__.py

def _q_getname (request, component):
    return RunUI(request, component)

class RunUI:
    _q_exports = ['details', 'check', ...]

    def __init__ (self, request, component):
        run_db = get_run_database()
        self.run = run_db.get_run(int(component)) 

    template _q_index (self, request):
        # /run/200/
        ... return index page ...

    template details (self, request):
        # /run/200/details/
        ... return a more detailed page ...

Form Framework

An optional part of Quixote that lives in the quixote.form subpackage, the framework lets us quickly implement basic (and not so basic) HTML forms.

Features:

The Form Class

Is a container for child widgets, and has three methods:

  1. render(): returns HTML for a Web page containing the form.
  2. process(): parses and checks user-supplied input values, flagging any errors.
  3. action(): finishes whatever action the form does (creating a new object, updating a database, whatever)

Widgets then produce HTML, and handle parsing of the resulting form element.

Example Form Code (I)

class LoginForm (Form):
    def __init__ (self, request):
        Form.__init__(self)
            
        self.add_widget(
            "string", "user_id",
            value=self.default_user_id,
            title="User ID",
            size=15, maxlength=32)
        self.add_widget(
            "password", "password",
            title="Password",
            size=15, maxlength=32)
        self.add_submit_button(name='login', value="Sign In")

    template render (self, request, action_url):
        title = "Sign In to the MEMS Exchange"
        standard.header(request, title, tree_info=[])
        for widget in self.widget_order:
            widget.render(request)
            "<br>"
            
        for button in self.submit_buttons:
            button.render(request)

        standard.footer(title)

Example Form Code (II)

    def process (self, request):
        form_data = Form.process(self, request)
        user_id = form_data["user_id"]
        password = form_data["password"]
        if user ID check fails:
            self.error['user_id'] = "Invalid user ID"
        elif not valid_password(user, password):
            self.error['password'] = "Invalid password"
        self.user = user
        return form_data

    def action (self, request, submit, form_data):
        request.session.set_user(self.user)
        self.user.record_login(request.get_environ("REMOTE_ADDR"))

Availability

These slides:
http://www.amk.ca/talks/

More on the MX toolset:
http://www.amk.ca/python/writing/mx-architecture/

ZODB/ZEO package:
http://www.amk.ca/zodb/

Grouch:
http://www.mems-exchange.org/software/grouch/

Quixote:
http://www.mems-exchange.org/software/quixote/

Join the quixote-users mailing list.