Testing the Multi-Resolution Volume Work

Cornerstone3D branch feat/multires-voxel-manager-base gives a volume more than one resolution of its voxels, and it lets a viewport draw the resolution that the device can hold. This page says how to run OHIF against that branch, and what to look at.

Read CS3D Integration Testing first. It describes the CS3D_REF mechanism and the local link scripts in full. This page adds only what is specific to the multi-resolution work.

What the branch changes

A volume of many images cannot always fit in one GPU texture. A series of 2464 images exceeds the limit of 2048 voxels on one axis that the GPU settings presets assume, so before this branch such a series could not be drawn at all on some devices, and the allocation failed.

The branch changes three things that a tester can see:

• A volume can hold several representations of its voxels. One is the data as the loader delivered it. Another is a box average of that data on a coarser grid.
• A viewport chooses the representation that the GPU settings can hold. The choice is made once, when the viewport adds its actor, and it is read from a GPU settings preset. The choice applies only with final downsampling on (lossyDownsampling: { final: true }). config/dev.js, config/netlify.js, config/e2e.js and config/customization.js turn it on. The other bundled configs, config/default.js and the Docker recipes among them, keep it off. Such a config turns it on only when the deployer sets lossyDownsampling.final, or when the cornerstone.gpuSettingsPreset customization selects a preset. See Which builds turn on final downsampling.
• A reduced representation follows the load. The loader delivers its frames over time, and each delivery redoes the part of the reduced data that the frame changed.

Which builds turn on final downsampling

Final downsampling is lossy, and it can cause problems on some devices. Thus only the developer builds turn it on. Cornerstone3D also keeps it off by default.

User requirements

• A developer build turns final downsampling on. The developer builds are the dev server (pnpm dev, pnpm dev:fast), viewer-dev.ohif.org, the Netlify deploy previews, and the e2e tests.
• A build that a deployment uses keeps final downsampling off, unless the deployer turns it on. These builds are pnpm build (config/default.js), the Docker recipes, and the other bundled deployment configs.
• The deployer can select a GPU settings preset with the cornerstone.gpuSettingsPreset customization.
• A developer build uses the medium preset, unless a ?customization=gpu/<preset> URL selects a different preset.
• With final downsampling off, the texture is the full-resolution texture, as before this work. Two other changes reach every build, also a build with final downsampling off:
  • The MPR load order changes. The coarse stages load first, and they copy early images over windows of up to 64 frames until the frames between them arrive.
  • The Cornerstone3D label "Transient Low Fidelity" shows in red while an MPR volume loads. The label stays on, because a user must know when a view does not show the full data. An OHIF indicator (OHIF/Viewers#6413) will replace it.

Implementation requirements

• config/dev.js, config/netlify.js, config/e2e.js and config/customization.js set lossyDownsampling: { final: true }.
• config/default.js and each config/docker-nginx-*.js do not set lossyDownsampling. A comment in each file shows how to turn it on, and lists the options.
• The other bundled configs do not set lossyDownsampling, and have no comment.
• config/dev.js, config/netlify.js and config/e2e.js set cornerstone.gpuSettingsPreset: 'medium' in the bootstrap phase of customizationService. A URL module applies its bootstrap phase after the app config, so the URL preset wins.
• The cornerstone extension (index.tsx) sets coarseInterleavedRetrieveStages as the retrieve stages of a volume, and hps/mpr.ts sets no imageLoadStrategy. Interleaved loading stays on, so the label shows during the load.

How OHIF reaches a reduced texture

The diagram gives the OHIF side of the sequence, with the default viewports of OHIF (genericViewports.enabled: false). Each CS3D n note names a sequence of the Cornerstone3D page Volume Render Sequences.

sequenceDiagram
    autonumber
    participant URL as ?customization=gpu/low-tablet
    participant Cust as customizationService
    participant Ext as cornerstone extension
    participant CS as Cornerstone3D
    participant HP as OHIFCornerstoneViewport
    participant Cache as CornerstoneCacheService
    participant CVS as CornerstoneViewportService
    participant Seg as segmentationService

    URL->>Cust: load customizations/gpu/low-tablet.jsonc
    Cust-->>Cust: cornerstone.gpuSettingsPreset = 'low-tablet'
    Ext->>CS: index.tsx: imageRetrieveMetadataProvider.add('volume', coarseInterleavedRetrieveStages)
    Ext->>CS: init.tsx: cs3DInit({ rendering: { lossyDownsampling } }) from appConfig
    Ext->>Ext: init.tsx: initGpuSettingsPreset({ customizationService, appConfig })
    Ext->>CS: setGpuSettingsPreset('low-tablet'), setLossyDownsampling({ final: true })
    Note over Ext,CS: CS3D 1. A later customization change applies the preset again,<br/>but only a viewport that adds an actor after the change reads it.
    Note over HP: hangingProtocolId=mpr gives three ORTHOGRAPHIC volume viewports
    HP->>Cache: createViewportData(displaySets)
    Cache->>CS: volumeLoader.createAndCacheVolume(volumeId, { imageIds })
    HP->>CVS: setViewportData(), then _setVolumeViewport(viewport, viewportData)
    Note over HP,CVS: mpr.ts has no imageLoadStrategy, so OHIF does not<br/>run a custom image load. The volume orders its own requests.
    CVS->>CS: volume.load()
    Note over CVS,CS: CS3D 3 and 4. The stages load, and each render fills dirty slices.
    CVS->>CS: setVolumesForViewport, then viewport.setVolumes(volumeInputs)
    Note over CVS,CS: CS3D 2. createVolumeActor, then resolveVolumeTexture<br/>chooses the full or the reduced texture set.
    opt the user loads a SEG display set
        Seg->>CS: createSegmentationForSEGDisplaySet(segDisplaySet)
        CVS->>Seg: addSegmentationRepresentation(viewportId, { blendMode })
        Seg->>CS: segmentation.addSegmentationRepresentations(viewportId, [labelmap])
        Note over Seg,CS: CS3D 6. The labelmap gets its own texture set,<br/>reduced by foreground majority.
    end

CornerstoneViewportService passes LABELMAP_EDGE_PROJECTION_BLEND when the viewport blend mode is MIP. That path builds a full-resolution CPU array, and it does not reduce. See sequence 6 of the Cornerstone3D page.

Running it locally

This repository states pnpm in the packageManager field of package.json, so every command uses pnpm. A command that starts with yarn fails with exit code 1 and the message This project is configured to use pnpm.

pnpm cs3d:checkout feat/multires-voxel-manager-base
pnpm cs3d:install
pnpm cs3d:build
pnpm cs3d:link

Then start OHIF with pnpm dev. To go back to the published packages, run pnpm cs3d:unlink.

A local run is necessary only to debug the code. For a test of the result, use the deploy preview (see Choosing a GPU settings preset).

Running it in CI

Add this line to the body of your pull request, outside every code block and every HTML comment:

CS3D_REF: feat/multires-voxel-manager-base

The branch must live in the cornerstonejs/cornerstone3D repository. A branch on a fork is rejected.

Choosing a GPU settings preset

A viewport reduces its texture to fit the GPU settings. To see a reduction on a device that needs none, select a smaller preset with the ?customization= URL parameter. The viewer ships one file for each preset under platform/app/public/customizations/gpu/:

Value	Texture edge	Texture memory	Use
gpu/low-tablet	256	1 GB	Almost every volume reduces. The reduction is easy to see.
gpu/low	256	2 GB	A weak integrated GPU. The same edge as low-tablet, with more memory.
gpu/medium	2048	8 GB	A device of the middle class.
gpu/high	2048	32 GB	The Cornerstone3D default. A device of the middle class can crash under it.
gpu/high-texture-4096	4096	32 GB	A control that fits no real device.

Use the deploy preview of the pull request for the tests. Each pull request gets a Netlify preview at https://deploy-preview-<number>--ohif-dev.netlify.app, and the CS3D_REF line of the pull request sets the Cornerstone3D branch of that preview. This example URL opens the 2464-image CT Body 0.5 CE on the preview of OHIF/Viewers#6295:

https://deploy-preview-6295--ohif-dev.netlify.app/viewer?StudyInstanceUIDs=1.3.6.1.4.1.14519.5.2.1.99.1071.24993177073256607564948872275593&hangingProtocolId=mpr&customization=gpu/low-tablet

For a local run, replace the host with http://localhost:3000.

Each file sets the cornerstone.gpuSettingsPreset customization. The app config must allow the prefix. config/netlify.js and config/dev.js set customizationUrlPrefixes, and config/default.js does not. Thus a build that serves config/default.js rejects the parameter, and the app stops at startup. To try a preset with config/default.js, set cornerstone.gpuSettingsPreset through customizationService in a copy of the config. You can also use a config that sets customizationUrlPrefixes.

gpu/high-texture-4096 states an edge of 4096, and no device is tested to render an edge above 2048. Some devices report a MAX_3D_TEXTURE_SIZE of 4096 or more, but a reported size is not a reliable limit. A real device can therefore fail to allocate the texture. The preset is the control of the demonstration: a series of up to 4096 images reduces nothing under gpu/high-texture-4096 and reduces under gpu/high, which shows that the reduction comes from the limit and not from a defect. Do not select the preset in production.

What to check

The viewer opens a large series. Load a CT series of more than 2048 images, for example Body 0.5 CE with the example URL above. Before this branch a device with a limit of 2048 could not allocate the texture. The series must now display.

A reformat holds no band and no black line. Look at the sagittal view and at the coronal view while the series loads. Each one must refine as the data arrives. A block of one image repeated down the view, or a black line across it, is a defect. Report it with the series and the number of images.

The measurements do not change. A viewport reads the full-resolution voxels for a measurement, whatever resolution it draws. A length, an area and a mean value must be the same as before the branch.

A segmentation still aligns with its images. A labelmap keeps its own resolution, and a reduced CT under it must not shift it.

See the reduction

The image alone does not prove which grid a viewport draws. Read the grid of the bound texture in the browser console. Select a viewport, then run:

const { cornerstoneViewportService, viewportGridService } = window.services;
const viewport = cornerstoneViewportService.getCornerstoneViewport(
  viewportGridService.getActiveViewportId()
);
viewport.getActors().map(({ referencedId, actor }) => ({
  referencedId,
  textureGrid: actor.getMapper().getScalarTexture?.()?.getGrid?.()?.dimensions,
}));

The result has one entry for each volume in the viewport. A loaded SEG adds a second entry for the labelmap. For the series Body 0.5 CE (512 × 512 × 2464), expect these grids:

Preset	textureGrid of the CT	Texture set
gpu/high-texture-4096	[512, 512, 2464]	full-resolution/full-extent
gpu/medium, gpu/high, or no customization with config/netlify.js	[512, 512, 1232]	reduced-1x1x2/full-extent/average
gpu/low, gpu/low-tablet	[256, 256, 247]	reduced-2x2x10/full-extent/average

A labelmap with the dimensions of the CT gets the same grid. The name of its set ends in foregroundMajority and not in average.

The ?debug=true stats overlay does not show the reduction. Its QUALITY row reads volume.getVoxelQuality() with no selector. That record describes the full-resolution data, and not the grid that the viewport draws.

The new viewports

Add &useNextViewports=true to a URL to draw through the generic viewports of Cornerstone3D. With the Cornerstone3D branch feat/multires-voxel-manager-base (cornerstonejs/cornerstone3D#2931, which now holds the acquisition-plane work of cornerstonejs/cornerstone3D#3018), the acquisition plane draws the original frames when the preset reduces the volume. The legacy viewports draw the acquisition plane from the reduced texture. Thus an axial view of the 2464-image CT under gpu/medium shows the average of two acquired slices on the legacy viewports, and one acquired slice on the new viewports.

An example URL on the deploy preview of OHIF/Viewers#6295:

https://deploy-preview-6295--ohif-dev.netlify.app/viewer?StudyInstanceUIDs=1.3.6.1.4.1.14519.5.2.1.99.1071.24993177073256607564948872275593&hangingProtocolId=mpr&customization=gpu/medium&useNextViewports=true

View	Legacy viewports	New viewports
Axial, the acquisition plane	The reduced texture. Lossy.	The original frames. Lossless after the frame loads. Until then the reduced texture shows, and the indicator shows lossy.
Sagittal and coronal	The reduced texture	The reduced texture, the same as legacy
Axial with a slab or the MIP blend mode	The reduced texture	The reduced texture

On the new viewports, check these items:

• A rotation from axial to sagittal and back keeps the slice, the pan and the zoom.
• The crosshairs and the sync of the views still work.
• A SEG stays aligned in each view. The axial view keeps the original frames while the labelmap shows.
• A small tilt of the axial view (for example with the crosshairs rotation) shows the reduced texture. The original frames show only when the tilt moves the plane by less than half a slice across the volume.
• For a tilt between about 0.5° and 8°, the progress scroll bar of the axial view has the volume form on the new viewports and the full form on the legacy viewports. The legacy viewports accept a tilt of up to about 8° as the acquisition plane.

What this branch does not do

The resolution of a viewport does not change while a volume loads. The device sets the resolution when the viewport adds its actor, and one resolution is used for the whole load. The values refine as the data arrives, so a reformat gets sharper, but the grid does not get finer. A ladder of grids is later work.

Nothing probes the device. A named preset is a set of values and not a measurement, so a deployment that selects a preset larger than the device holds gets an allocation failure. A later pull request will add an automatic mode that probes the device.

Known limitations

• Only the mpr protocol uses the coarse stages. fourUp, main3D, mprAnd3DVolumeViewport, only3D, primary3D, primaryAxial and the tmtv protocol set an imageLoadStrategy, and a protocol with a strategy does not use the retrieve stages of the volume. These protocols still show gray lines while a volume loads.
• The Cornerstone3D fidelity indicator can cover the OHIF menus. The indicator (fi-root) has a z-index of 1000, which is above the OHIF menu layer. On a reduced volume the indicator can take a click on a menu item. The OHIF fidelity indicator, OHIF/Viewers#6413, will address this.
• No automatic test covers the new viewports or the reduction. See OHIF/Viewers#6404.

Where the design is written

The Cornerstone3D repository holds the design of this work. Read the pages in this order:

1. Volume Render Strategy describes how a viewport builds its strategies, how one render chooses among them, and how a derived representation follows the load. Its section "Where the code is" gives the file of each piece.
2. Volume Render Sequences gives the sequence of calls as diagrams, from the preset to the fill of a reduced texture, and for a segmentation.
3. Composite Voxel Manager describes how a volume holds more than one representation of its voxels.