Instrument data preprocessing

After a watcher uploads a raw file, Data Hub can run a preprocessor on it: code that reads the vendor’s file, pulls out settings like wavelength or imaging mode, and builds something you can look at in the browser. Seven instrument types have one today. This page covers what each produces, why a file sometimes gets no processing, and what adding a new instrument involves.

How a file reaches a processor

1. A file arrives in raw storage, which triggers the Lambda. The bucket notifies on every upload, with no per-instrument filtering.
2. The Lambda checks the filename against a filename gate, a cheap test that skips files no processor wants before making any API call.
3. It reads the instrument, then looks up a processor by the instrument’s type. Unmapped types stop here.
4. The processor downloads the file, extracts metadata, writes processed files when it has any, and updates the run through the API.
5. The file’s status moves from processing to completed or failed.

Why an upload sometimes produces nothing

Two silent no-ops explain a file that uploaded and then produced nothing. Neither marks the file failed, because nothing went wrong:

• The filename missed the gate. Each type accepts a narrow pattern, listed in the table below. An Azure Cielo run only triggers on files ending _cq values.csv, so the other CSVs a qPCR run writes are stored and ignored.
• The instrument has no type set, or a type with no processor. Files upload and stay browsable; you get no metadata and no reports.

Filename gates apply to uploads only. Choosing Reprocess in the web app skips the gate, because clicking it states your intent. That’s why a file that produced nothing on upload can still process when you reprocess it by hand.

Set the instrument type

Processing is chosen by the instrument’s type, not by its instrument ID. The ID (azure-cielo-qpcr) names one physical machine. The type (qpcr) names a vendor’s file format, and several instruments can share one.

To set it, open the instrument page, choose Edit from the three-dot menu, and pick a type.

Supported instrument types

Agilent 4150 TapeStation

Reads the tape type out of peak-table CSV filenames and shows TapeStation reports on the run page.

• Expected files: CSVs named like YYYY-MM-DD - HH-MM-SS-<TapeType>_peakTable.csv, plus matching PDFs
• Metadata: Tape Type, for example gDNA or HSD1000, when the filename matches that pattern
• Processed files: none
• On the run page: tape-type badge, a PDF viewer, and peak-table reports

Uploads trigger on the PDF, not the CSV. A run of CSVs alone stays unprocessed until you reprocess it by hand.

Cytiva ÄKTA FPLC

Stores ÄKTA chromatography PDFs so you can read them in the browser. The processor reads nothing out of the PDF itself yet, so this type gets no badges or filters.

• Expected files: .pdf reports from the ÄKTA
• Metadata: none
• Processed files: none
• On the run page: files table, plus each PDF embedded under Report Data

A newly seeded FPLC stays generic on purpose until an operator has checked its output format.

Azure 600 Gel Doc

Rebuilds each gel TIFF as a viewable image and reads the imaging settings the instrument recorded.

The processor stretches intensities to the 1st to 99th percentile, per RGB channel or as grayscale, then lays every page of the TIFF side by side in one 300 dpi PNG. The stretch is cosmetic: it makes faint bands visible on screen and doesn’t change the raw file.

• Expected files: TIFF from the Azure 600
• Metadata: capture_type, imaging_mode (true color, fluorescence, or chemiluminescence), wavelengths, colors, read from the TIFF XPComment tag
• Processed files: contrast-enhanced PNG
• On the run page: imaging badges, files table, image viewer for the PNGs

Those metadata fields also drive the column filters on the instrument page.

Azure Cielo qPCR

Reads dye channel names so you can filter and badge runs by channel.

• Expected files: Cq Values CSV. Other files in the run upload normally but contribute no metadata
• Metadata: dye_channels, the unique names in the Fluorescence column
• Processed files: none
• On the run page: dye-channel badges and report tables

Epson V700 Scanner

Finds agar plates in a flatbed scan and measures the colonies on them.

The processor looks for plates sitting in gold 3D-printed frames. For each plate it finds, it crops the plate, then detects colonies: crop the edge, compare against the plate background, apply a difference-of-Gaussians filter, threshold, and measure whatever survives. Measurements come out in physical units, so colony areas are comparable between scans.

• Expected files: TIFF plate scans
• Metadata: dpi, color_mode, plate_count, plate_boxes, plus colony_detection summaries when colonies were measured
• Processed files: a web-sized JPG with plate outlines and colony boxes drawn on, and a _colonies.csv of per-colony area, centroid, eccentricity, and mean RGB
• On the run page: scanner badges, files table, and reports for the processed CSV

When no plates are found, you still get a resized JPG of the whole scan, and colony detection is skipped. An empty plate_count usually means the frames weren’t recognized rather than that the plates were blank.

Hina Microscope

Turns each Nikon ND2 into a single composite image you can view in the browser.

Every channel is reduced to one 2D frame (a maximum projection over Z, the first index over time or position), stretched to the 1st to 99th percentile, then composited onto a brightfield background in each channel’s fluorophore color.

• Expected files: ND2
• Metadata: sizes, channels (name, excitation and emission wavelength, color), and dimensions
• Processed files: an RGB overlay JPG for every file in the run
• On the run page: channel and size badges, files table, image viewer for the JPGs

Channel layout is identical across a Hina run, so metadata is written once by whichever file finishes processing first. Later files still get their JPG and skip the metadata write. Reprocessing one file therefore won’t correct run metadata if the layout changed mid-run.

SpectraMax iD3 and iD5 plate readers

Both models share one processor, which converts SoftMax Pro exports into the well data behind the interactive plate maps.

• Expected files: SoftMax .xls, which is UTF-16 text with a Plate: header rather than a real Excel workbook
• Metadata: measurement_mode (for example Absorbance), measurement_type (for example Endpoint or Kinetic), and wavelengths as bare numbers with no unit suffix
• Processed files: {run_id}_raw_well_data.csv
• On the run page: wavelength, mode, and type badges, plus plate maps built from that CSV

Types with no processor

generic and instant_raman have no processor, and neither does an instrument whose type is unset. Those uploads still work: files are stored, and you can browse, search, claim, comment on, and download them. What you don’t get is extracted metadata, badges, filters, or processed artifacts.

Leaving a new instrument as generic is the safe default. It keeps data flowing while someone confirms which processor, if any, can read its files.

When processing fails

A file with status failed carries an error message on the run page. Open the run, then choose Reprocess on that file, or on the whole run where that action is available. See Reprocess or delete a run.

Reprocessing is also the fix after an engineer corrects a parser: the code changed, but the stored result didn’t, until you ask for it again.

Add support for a new instrument

Adding an instrument type is a development task spanning the web app and the Lambda, not a configuration change an admin can make in the dashboard. Budget for a database migration and a Lambda image deploy.

You can exercise a processor locally without S3 or API access using the data-hub-process CLI, and drive the whole handler against a local S3 mirror with its handler subcommand.

Full detail, including the archive builder and the Docker build, is in the Lambda developer docs.