2.4.1   Requirements on Services¶

• DMS-REQ-0380 (Priority: 1b), HiPS Service: “The Data Management system shall include a secure and authenticated Internet endpoint for an IVOA-compliant HiPS service. This service shall be advertised via Registry as well as in the HiPS community mechanism operated by CDS, or whatever equivalent mechanism may exist in the LSST operations era.”
• DMS-REQ-0381 (Priority: 2), HiPS Linkage to Coadds: “The HiPS maps produced by the Data Management system shall provide for straightforward linkage from the HiPS data to the underlying LSST coadded images. This SHOULD be implemented using a mechanism supported by both the LSST Science Platform and by community tools.”
• DMS-REQ-0384 (Priority: 1b), Export MOCs As FITS: “The Data Management system shall provide a means for exporting the LSST-generated MOCs in the FITS serialization form defined in the IVOA MOC Recommendation.”
• DMS-REQ-0340 (Priority: 2), Access Controls of Level 3 Data Products: “All Level 3 data products shall be configured to have the ability to have access restricted to the owner, a list of people, a named group, or be completely public.” - Note that Level 3 data products may include user-generated images.
• DMS-REQ-0160 (Priority: 1b), Provide User Interface Services: “The DMS shall provide software for User Interface Services, including services to: browse LSST data products through astronomical views or visualizations; create and serve ”best” images of selectable regions of the sky; resample and re-project images, and visualize catalog content.” - We assume that, in line with the Science Platform Vision, LSE-319, these services must include programmatic Web APIs.
• DMS-REQ-0298 (Priority: 1a), Data Product and Raw Data Access: “The DMS shall provide software for Data Access Services to list and retrieve image, file, and catalog data products (including raw telescope images and calibration data), their associated metadata, their provenance, or any combination thereof, independent of their actual storage location.”
• DMS-REQ-0065 (Priority: 1b), Provide Image Access Services: “The DMS shall provide a service for Image Access through community data access protocols, to support programmatic search and retrieval of images or image cut-outs. The service shall support one or more community standard formats, including the LSST pipeline input format. Discussion: At least the FITS image format will be supported though an IVOA-standard service such as SIAP. Other image formats such as JPG may be more compatible with education/public outreach needs.”
• DMS-REQ-0387 (Priority: 1b), Serve Archived Provenance: “The Data Management System shall make the archived provenance data available to science users together with the associated science data products.”
• DMS-REQ-0311 (Priority: 1b), Regenerate Un-archived Data Products: “The DMS shall be able to regenerate unarchived data products to within scientifically reasonable tolerances. Discussion: Unarchived data products currently include Processed Visit Images for single visits, some Coadds, and Difference Images. Scientifically reasonable tolerances means well within the formal uncertainties of the data product, given the same production software, calibrations, and compute platform, all of which are expected to change (and improve) during the course of the survey.”
• DMS-REQ-0336 (Priority: 1b), Regenerating Data Products from Previous Data Releases: “The DMS shall be able to regenerate data products from previous data releases to within scientifically reasonable tolerances.”
• DMS-REQ-0127 (Priority: 2), Access to input images for DAC-based Level 3 processing: “The DMS shall provide access to all Level 1 and Level 2 image products through the LSST project’s Data Access Centers, and any others that have been established and funded, for Level 3 processing that takes place at the DACs.”

2.4.2   Performance Requirements¶

• DMS-REQ-0375 (Priority: 2), Retrieval of postage stamp light curve images: “Postage stamp cutouts, of size postageStampSize [51 pixels] square, of all observations of a single Object shall be retrievable within postageStampRetrievalTime [10 seconds], with postageStampRetrievalUsers [10] simultaneous requests of distinct Objects.”
• DMS-REQ-0374 (Priority: 1b), Retrieval of a PVI from a single CCD: “A Processed Visit Image of a single CCD shall be retrievable using the VO SIAv2 protocol within pviRetrievalTime [10 seconds] with pviRetrievalUsers [20] simultaneous requests for distinct single-CCD PVIs.”
• DMS-REQ-0376 (Priority: 1b), Retrieval of focal-plane visit images: “All Processed Visit Images for a single visit that are available in cache, including mask and variance planes, shall be identifiable with a single IVOA SIAv2 service query and retrievable, using the link(s) provided in the response, within allPviRetrievalTime [60 seconds]. This requirement shall be met for up to allPviRetrievalUsers [10] simultaneous requests for distinct focal-plane PVI sets.”
• DMS-REQ-0377 (Priority: 1b), Retrieval of a CCD-sized image from a coadd: “A CCD-sized cutout of a coadd, including mask and variance planes, shall be retrievable using the IVOA SODA protocol within ccdRetrievalTime [15 seconds] with ccdRetrievalUsers [20] simultaneous requests for distinct areas of the sky.”
• DMS-REQ-0373 (Priority: 2), Retrieval of focal-plane-sized images: “A 10 square degree coadd, including mask and variance planes, shall be retrievable using the IVOA SODA protocol within fplaneRetrievalTime [60 seconds] with fplaneRetrievalUsers [10] simultaneous requests for distinct areas of the sky.”

2.4.3   Select Requirements on Types and Content of Images (assembled here for context)¶

• DMS-REQ-0326 (Priority: 2), Storing Approximations of Per-pixel Metadata: “Image depth and mask information shall be available in a parametrized approximate form in addition to a full per-pixel form.”
• DMS-REQ-0024 (Priority: 1a), Raw Image Assembly: “The DMS shall assemble the combination of raw exposure data from all the readout channels from a single Sensor to form a single image for that sensor. The image data and relevant exposure metadata shall be integrated into a standard format suitable for down-stream processing, archiving, and distribution to the user community.”
• DMS-REQ-0068 (Priority: 1a), Raw Science Image Metadata: “For each raw science image, the DMS shall store image metadata including at least: i) Time of exposure start and end, referenced to TAI, and DUT1; ii) Site metadata (site seeing, transparency, weather, observatory location); iii) Telescope metadata (telescope pointing, active optics state, environmental state); iv) Camera metadata (shutter trajectory, wavefront sensors, environmental state); v) Program metadata (identifier for main survey, deep drilling, etc.); and vi) Scheduler metadata (visitID, intended number of exposures in the visit).
• DMS-REQ-0069 (Priority: 1a), Processed Visit Images: “The DMS shall produce Processed Visit Images, in which the corresponding raw sensor array data has been trimmed of overscan and corrected for instrumental signature. Images obtained in pairs during a standard visit are combined. Discussion: Processed science exposures are not archived, and are retained for only a limited time to facilitate down-stream processing. They will be re-generated for users on-demand using the latest processing software and calibrations. This aspect of the processing for Special Programs data is specific to each program.”
• DMS-REQ-0010 (Priority: 1b), Difference Exposures: “The DMS shall create a Difference Exposure from each Processed Visit Image by subtracting a re-projected, scaled, PSF-matched Template Image in the same passband. Discussion: Difference Exposures are not archived, and are retained for only a limited time to facilitate Alert processing. They can be re-generated for users on-demand.”
• DMS-REQ-0334 (Priority: 1b), Persisting (Level 2) Data Products: “All per-band deep coadds and best seeing coadds shall be kept indefinitely and made available to users. Discussion: This requirement is intended to list all the data products that must be archived rather than regenerated on demand. DMS-REQ-0069 indicates in discussion that Processed Visit Images are not archived. DMS-REQ-0010 indicates in the discussion that Difference Exposures are not archived.”
• DMS-REQ-0279 (Priority: 1b), Deep Detection Coadds: “The DMS shall periodicaly create Co-added Images in each of the u,g,r,i,z,y passbands by combining all archived exposures taken of the same region of sky and in the same passband that meet specified quality conditions.”
• DMS-REQ-0280 (Priority: 1b), Template Coadds: “The DMS shall periodically create Template Images in each of the u,g,r,i,z,y passbands that are constructed identically to Deep Detection Coadds, but where the contributing Calibrated Exposures are limited to a range of observing epochs templateMaxTimespan [1 year] the images are partitioned by airmass into multiple bins, and where the quality criteria may be different.”
• DMS-REQ-0281 (Priority: 1b), Multi-Band Coadds: “The DMS shall periodically create Multi-band Coadd images which are constructed similarly to Deep Detection Coadds, but where all passbands are combined.”
• DMS-REQ-0330 (Priority: 2), Best Seeing Coadds: “Best seeing coadds shall be made for each band (including multi-color).”
• DMS-REQ-0335 (Priority: 1b), PSF-Matched Coadds: “One (ugrizy plus multi-band) set of PSF-matched coadds shall be made but shall not be archived.”
• DMS-REQ-0338 (Priority: 2), Targeted Coadds: “It shall be possible to retain small sections of all generated coadds.”
• DMS-REQ-0106 (Priority: 1b), Coadded Image Provenance: “For each Coadded Image, DMS shall store: the list of input images and the pipeline parameters, including software versions, used to derive it, and a sufficient set of metadata attributes for users to re-create them in whole or in part.”
• DMS-REQ-0132 (Priority: 1a), Calibration Image Provenance: “For each Calibration Production data product, DMS shall record: the list of input exposures and the range of dates over which they were obtained; the processing parameters; the calibration products used to derive it; and a set of metadata attributes including at least: the date of creation; the calibration image type (e.g. dome flat, superflat, bias, etc); the provenance of the processing software; and the instrument configuration including the filter in use, if applicable.” - We assume that this information is not only to be recorded but also made available to users.
• DMS-REQ-0329 (Priority: 2), All-Sky Visualization of Data Releases: “Data Release Processing shall generate co-adds suitable for use in all-sky visualization tools, allowing panning and zooming of the entire data release.”
• DMS-REQ-0379 (Priority: 1b), Produce All-Sky HiPS Map: “Data Release Production shall include the production of an all-sky image map for the existing coadded image area in each filter band, and at least one pre-defined all-sky color image map, following the IVOA HiPS Recommendation. Discussion: The maximum resolution of the image maps is TBD; however, it would be desirable for it to be at least close to the underlying coadded image resolution, in order not to give a poor impression of the data quality. It is possible that the highest-resolution HiPS tiles could be provided on-demand from the LSST cutout service. […]”

3.1.1   Current Science Pipelines considerations relating to coadds¶

3.2.1   Compressed Processed Visit Images¶

It was realized early in the life of the LSST project that retaining the fully-calibrated single-epoch images, known as Processed Visit Images (PVIs), would be one of the largest single contributions to the storage capacity budget for the project. In the preliminary-design era, then, it was decided to plan for a tradeoff of computation against storage space, baselining a capability to recreate PVIs on-demand from the (much smaller) raw data. (The raw images are smaller because a) they are missing variance and mask planes, and b) they have an integer pixel data type and are generally more suitable for lossless compression as a result.)

Subsequently, after the start of construction, the team recognized the existence of common use cases from the transient and variable phenomena science themes that involve requests for stacks of cutouts following an object through time in the survey. Such requests could not have been addressed in a timely way if all the single-epoch images had to be re-created. Under RFC-325 a proposal was adopted to add the storage of lossy-compressed PVIs to the baseline, with image services to match.

At a technical level, the current tentative approach to this is to define a new Butler dataset type, calexp_comp, which represents PVIs processed through astronomy-quality lossy compression (calexp is the dataset type for the PVIs themselves). Rubin/LSST image services will be able to return both native and compressed PVIs, and perform cutouts on either type. Image metadata services will treat them as separate datasets differing primarily only in their dataproduct_subtype and their size.

3.2.2   Processed Visit Images¶

3.2.3   Raw Data¶

Raw image data is proposed to be available in two forms: the original files materialized at the time of data acquisition, and files with the original raw pixel data but updated metadata (particularly WCS) determined at the time of pipeline processing. It is not clear whether the updated-metadata files would be materialized or created on demand.

4.1.1   Pixel-preserving cutouts of explicitly specified images¶

Given a specific image ID, or list of image IDs, identified from an image metadata service (or in some other way - the same identifiers must be available in the context of retrieving an image from the Butler) - in this use case the user will wish to extract a subregion of the image, preserving exactly the pixel values and coordinates. No reprojection or rescaling is involved. Metadata such as the WCS and PSF are expected to be preserved as accurately as possible. This use case is equally applicable to raw, PVI, compressed-PVI, diffim, and coadded images. Applied to coadded images, it is only an example of this use case if the ID(s) of a patch-level image (or images) is (are) supplied. It should be capable of being run against both persisted and non-persisted (“virtual”) data products, recognizing that if re-creation of a non-persisted data product is involved, the response latency will be longer.

4.1.2   Reprojected/resampled cutouts of explicitly specified images¶

In this use case, the output image(s) is (are) resampled to a different grid than the input. For example, this could arise when assembling a “filmstrip” of cutouts from PVIs across the entire survey for a specific point on the sky, and projecting them all into a “north-up” pixel grid. Other uses might include rebinning to a coarser pixel scale, or projecting PVIs onto a tangent plane. In this use case, as in the previous one, the image(s) must be specified via explicit single-epoch, single-CCD IDs (typically visit IDs), or explicit coadd patch IDs.

4.1.3   Mosaics from single visits¶

In this use case, the user is interested in coverage of a larger region on the sky from a single visit than a CCD, or a region that overlaps a boundary between CCDs or rafts. Note that the inter-CCD and inter-raft gaps are in no way guaranteed to be integer numbers of pixels, or even precisely parallel, so mosaics crossing gaps will have discontinuities in their WCS unless they are reprojected. It may be useful to provide an option to the user to choose between reprojection onto a tangent plane vs. pixel-preserving mosacing into a “nominal” geometry with parallel, integer gaps; this would be outside the standard SODA option space.

4.1.4   Arbitrary optimal cutouts from coadds¶

4.1.5   Generation of preview graphics¶

In each of the above use cases, the user will have the option of requesting a preview image in a commonly available format (e.g., PNG) instead of the standard FITS serialization.

5.1.1   Endpoint services¶

Image metadata query:

ObsTAP, from ObsCore 1.1 or later (as an initial priority), with SIAv2 (also ObsCore-based and serving equivalent data) later

Catalog query:

TAP 1.1 or later

Image cutout service:

SODA 1.0 or later

DataLink links service:

DataLink 1.0 or later (very likely to be revised in the near future)

Note that there is no general “image service” per se in the current IVOA model. SIAv2 (and its predecessor) are image metadata services, operating on a model of queries returning URLs to the actual images (possibly indirected through DataLink links services). The endpoints described by those URLs need not provide anything other than static file service. The SODA standard covers the specific case of cutouts (in general not previously existing) from identified datasets which themselves may or may not be materialized or statically accessible.

5.1.2   Supporting standards¶

Service endpoints, self-description, and status:

VOSI 1.1 or later

Service parameters and datatype definitions:

DALI 1.1 or later

Asynchronous service interfaces:

UWS 1.1 or later

Query language:

ADQL 2.00 or later (ADQL 2.1 is imminent)

5.1.3   Extensibility¶

Where these standards fall short of meeting the needs of the project:

• Many of them have specified avenues for enhancing a service with additional metadata or parameters. Examples:
  • A TAP/ObsTAP service can supply additional information in its TAP_SCHEMA schema;
  • A DataLink links service can provide publisher-specific values for the semantics column; and
  • An ObsCore table may contain a) optional ObsCore-defined columns, and/or b) additional service-specific columns.
• The project is free to develop service protocols and data formats of its own. However, consistent with the “VO-first” plan, we will strive to respect the layering of standards. For example, if the creation of a non-IVOA-standard service is required in order to fulfill a need, it should still be compliant with DALI, VOSI, and, if applicable, UWS. This policy facilitates service concepts such as the use of DataLink to direct users to these non-standard services.

5.1.4   Service Registration¶

The present edition of this document does not address how Rubin/LSST services will be represented in the IVOA Registry/ies, nor does it address whether the project will operate its own registry.

5.3.1   WCS Metadata¶

Image files produced in the FITS format by the project will contain FITS-standard WCS information. For coadded images, which are warped to a precise pixel grid, a trivial TAN projection will be used. For FITS tiles in the HiPS image maps produced by the project, the HPX projection will be used.

For single-epoch images where the actual Camera WCS must be modeled, the full-precision WCS model produced by the pipeline will not be completely captured by FITS-standard WCS headers. Full-precision WCS data will be included in an extension in the image file containing WCS information, using the ASDF model also used by JWST, represented as YAML ASCII string data in a binary table extension.

6.2.1   CAOM2¶

6.2.2   ObsCore¶

8.1.1   Minimal (Frozen) Portal¶

8.1.2   Aspirational Portal Design¶

10.1.1   Generic DALI/VOSI service implementation framework¶

The service concept described above involves the creation of a variety of services, some directly conformant to application-level IVOA standards such as SODA and DataLink, and some unique to Rubin but conformant to the framework standards such as DALI and VOSI. In order to ease the development, deployment, and maintenance of these services, we envision the creation and/or re-use of a framework for supplying these common elements of a service implementation.

Several of the envisioned services (SODA/cutouts, virtual-data-product re-creation, forced photometry on demand) ultimately require the invocation of Rubin Science Pipelines stack code, with its Python interfaces. Hence we would need either a service framework actually implemented in Python, or one implemented in another language such as Java or its descendants which is then able to invoke a “payload” of Python code.

The service framework should provide, at a minimum:

• skeleton implementations of /availability and /capability endpoints, with an extensible interface for supplying service-application-specific behavior where needed; and
• definition and parsing of a URL-with-query-parameters API, following DALI standards, including the parsing of DALI-compliant parameter value types and their translation into Python objects, ideally Astropy objects where available.

Desirable features would include:

• the ability to generate DataLink-compatible service descriptors from the API parameter definitions;
• the ability to either generate, or start from, an OpenAPI v3 service description, enabling the provision of “Swagger” UI pages for users to explore the APIs’ capabilities, and perhaps in the future to facilitate the provision of metadata-driven widgets in the Notebook Aspect for access to the APIs.

For an example of the use of the OpenAPI specification and Swagger to deliver an elementary UI, see the CADC TAP service page at https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/argus/ and the SIA service page at https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/sia/ . The value of this for user education and for operations-team debugging activities is unmistakable. There is a cost in providing interfaces of this sort, but it is greatly mitigated by building the necessary apparatus in a generic way.

Some aspects of this can likely be obtained on the Java side by the adoption of layered elements of the CADC code base. However, the propagation of DALI parameters to Python code that actually performs computational work is an important part of the vision. On this side, there is, at the time of writing, no fully satisfactory open-source Python framework with this features. It would be immediately useful internally, as well as a valuable community service, if the Rubin project were to produce such a framework.

Such a framework would be designed to minimize the “boilerplate” work on API parsing and DALI/VOSI-conformance associated with delivering a new service.

10.1.2   UWS-based asynchronous services¶

A number of the services that the Science Platform will require will be natural candidates for implementation as asynchronous services.

10.1.3   VOTable annotation¶

10.1.4   TAP (and ObsTAP)¶

The TAP service planned for use in the Science Platform is the CADC-derived, Java-based one which is already in use in the prototype/demonstration-oriented Science Platform deployed at NCSA. Thus far the CADC code base has proven valuable. It was relatively easily adapted to the Qserv back end, and the relationship between the Rubin project and CADC is positive and has allowed upstreaming a number of changes.

The Rubin project has not yet attempted to use the currently limited capabilities of the CADC framework for DataLink service descriptor or <GROUP> annotation of VOTable query results. Because DataLink is an important part of the service concept described in this note, we may find that it is an area in which it is valuable for us to make a substantial upstream contribution.

10.1.5   SIAv2¶

As also noted in the main body of this document, CADC has an implementation of SIAv2 which performs virtually all of the actual work of performing a query by deferring to an underlying ObsTAP service. This is feasible because of the close alignment of the ObsCore (which contains ObsTAP) and SIAv2 standards, in which almost all SIAv2 query capabilities are expressible as ADQL against the ObsCore data model. We should investigate whether the CADC implementation is in fact practical for use against Rubin’s specific ObsTAP service.

Note that an implementation of SIAv2 as a call-through to ObsTAP would then directly benefit from any progress on DataLink and/or <GROUP> annotations in the VOTable output from the ObsTAP service.