In attendance:
Roland Bacon bacon@obs.univ-lyon1.fr Petra BoehmVeronique Cayatte veronique.cayatte@obspm.fr Yannick Copin copin@strw.leidenuniv.nl Eric Emsellem emsellem@obs.univ-lyon1.fr Pierre Ferruit pierre@obs.univ-lyon1.fr Rachel Johnson raj@ast.cam.ac.uk Markus Kissler-Patig mkissler@eso.org Vincent le Brun vlebrun@astrsp-mrs.fr Martin Roth
Juergen Schmoll jurgen.schmoll@durham.ac.uk Jeremy Walsh jwalsh@eso.org (Chair)
The meeting took place in the old library surrounded by historic volumes. Jeremy Walsh opened the meeting and stressed that the WG was entering a second era now that the Research Training Network application had been completed. He thanked Martin Roth especially for his huge effort in pulling the proposal together and chasing us all to the deadline. Initial feedback on the pre-submission of the proposal makes us hopeful that it will be accepted. This meeting was dedicated to finding a common 3D spectroscopy format. This is important both for effective interchange of data among ourselves but also as the first building stone of the RTN.
Euro3D proposal
Martin Roth had not yet distributed copies to all of the complete proposal.
There was a question whether Opticon might pay for colour copies but colour
was not felt to be important. Martin summarized by saying that training was
seen as a vital component of the application. The pre-submission had resulted
in a very useful telephone conversation with the EU. It was stated that
a list of milestones were missing, and these were later added. The major
concern was felt to be the 3 year duration of the RTN.
Proposals are currently being evaluated and the accepted proposals should be announced by November. Then negotiations begin on the contract. February 2002 is seen as the earliest possible date at which the contract could begin. There was discussion about finding post-docs for the RTN positions. It is good to begin thinking now as the typical post-docs come available around September. Martin was concerned that the visualization task with a total of 51 person months would be highly distributed and yet be a vital part of Euro3D. In the contract negociation the deliverables would have to be carefully stated but without being unrealistic. It may be possible to negociate a 4th year extension to the contract.
There was a round-the-table to determine if anyone knew of students or soon post-docs who would be willing/able to be employed by the RTN. The International Max-Planck Research School on Astrophysics in Munich was seen as a potential recruiting ground. A person must have been 6 months out of a country before he can be recruited back. ESO is considered as being German in this regard. Some tricks are possible with having students at ESO for a while. Some names were mentioned around the table. The probable duration of the RTN would be mid-2002 to mid-2005. All are asked to keep their eyes open for possible recruits. An article in the ESO Messenger when the RTN acceptance is announced was suggested to advertise the project. Aim is the December 2001 ESO Messenger.
3D Format
The discussions began by presentations of the range of options:
2D+Table
2D+Cube or cube+cube
raw 3D cube
2D+Table
Pierre Ferruit presented the pros and cons of 2D+Table format.
The 2D image has one spectrum per row. Different images can hold
the signal, the variance and the data quality. This format is very
flexible and can contain any MOS or IFU setup. It has the advantage
of a spectrogram-like approach with the spectra spatially contiguous
on the display, a mode familiar to long slit users. Another advantage
is that the whole dataset can be visualised at once since it is 2D.
The clear disadvantage is that the spectra must be reconstructed to make
images on the sky.
It was agreed that the 3D format is for data where the instrument
signature is removed. This followed Rachel Johnson's comment that CIRPASS
had spectral fibres which did not have a spatial correspondence,
used for checks on stray light, etc.
The accompanying table can be FITS with one row per spectrum. Then
there is flexibility as to what information is held in columns. Tools
exist to access and handle such simple tables in most packages which is
not true for 3D FITS tables. Again processing has to be done to
reconstruct images on the sky by look-up in the table. Whilst some
integral field instruments have regular grids (e.g. MPFS) others
do not (e.g. Integral).
In the ensuing comments there were questions about storing mosaiced
data. For cube formats this could involve many unused pixels. There
was a concern however that in a few years all the new IFU's would have
regular grids.
Rachel reported on the format adopted by CIRPASS. Hexagonal pixels are employed. A 2D + Table format is used with 1 row per spectrum and the row number corresponding to the lens number. The format, which is similar to COHSE and SPIRAL, is multi-extension FITS with a primary header, the 1st extension the table, 2nd extension science data, 3rd extension variance and 4th extension data quality.
2D+Cube
Yannick Copin presented the 2D image+cube (with cube+cube as a variant)
based on his excellent document distributed beforehand
Postscript. As opposed to the
necessity of reconstructing the spatial dimension using the spatial
imformation of the table in the 2D+Table case, the cube contains the
spatial information by reconstructing the spectral information. The spectral
information is held as 2D images similar to the 2D+Table format
with its advantages. This
data can be stored according to position in a spatially oriented 2D
table. The storage of the derived spatial quantities by the cube can
be flexible, eg. flux, velocity, etc. However it was noted in the
discussion that the FITS standard does not allow differing quantities
to be stored on a single NAXIS; so can't mix slices of flux, velocity,
etc in a single cube. The format requires a regular
sampling in each direction but not necessarily the same to accommodate
square and hexagonal packing.
The format emphasizes the spatial directions of the cube but gives
an image directly only for square pixels. If spatial data is missing
then the simple z-slice extraction is not enough, or has to have
holes. For hexagonal packing each slice of the cube must be resampled
to produce the sky image. The format is not suited to cases of
non-regular spatial binning (e.g. from adaptive pixel reconstruction).
The 2D image could easily be replaced by a cube, with data stored
in such a way that for regular square pixels grids a simple slice
gives a sky image.
3D Cube
Markus Kissler presented the "raw" 3D cube format. This works only for
square pixels on a regular grid. The great advantage is that any z
slice is an image on the sky so can be thought of as "natural".
However for other grids data must be rebinned and it cannot always be
assumed that data has or can be rebinned.
All agreed that a simple cube with ortho-normal alpha,delta,lambda is
out.
Martin suggested that by transposing the cube axes one could store spectra in a way that facilitated sky map formation: e.g. 1st axis lambda, 2nd alpha, 3rd dec. But again this would only apply to regular filling.
IFU Discussion Board
Juergen Schmoll briefly presented Alasdiar Allan's IFU Discussion Board. This consisted of six people from Durham, STARLINK, NOAO, Gemini South and Cambridge exchanging views on topics from IFU meetings, datacube or not, FITS extensions, test data and Euro3D. There are reports on a STARLINK IFU Software meeting at Edinburgh (27/4/2000) at which various FITS multi-extension formats were discussed. Data quality arrays were discussed and an IRAF datacube format. Last entries are in January 2001. It is clear that the 3D cube / 2D+Table divide is strong. Worth looking through.
Discussion
There was a consensus that it is the spectra that really hold the science information and that the maps are generated from the spectra. The physics is in the spectra and it is this that should be primary in our considerations. The larger formats being discussed for future IFU's (up to 1000x1000 spatial pixels) would produce data which would be beyond simple visualization even as stacked 1D spectra.
All agreed that science related quantities SHOULD be allowed to be stored in the format.
The Oasis software had two modes for memory use depending on whether the approach was slice oriented (sky images) or spectrum oriented.
There was a vote and all were unanimous that the spectra should be stored in a stacked image with NAXIS=2.
A comment was made that there is power and saving by attaching to existing software, eg. for FITS tables handling.
The remaining discussion centred on how to store the spatial information and the science related quantities. One possibility was a 3D FITS table for the science related quantities. This is not allowed under the FITS standard. The alternative is to have separate 2D tables for each related quantity.
Whatever is provided there should be converters available for IRAF and IDL.
Markus said that Midas is effectively dying as no more development will take
place.
During a coffee break Martin ran some tests on his laptop reading synthetic
VIMOS IFU data as cubes and 2d+tables and showed that I/O and formatting
into x,y,lamda arrays are not a problem. Tests were run in IDL.
It was decided that the spatial information: X,Y position of each lens be stored in a single table with header keyword information to handle size of lenses, shape, alpha, delta of array centre, etc. In the case of fibres without a connection to the lenslet array (e.g. CIRPASS), they could be given exceptional positions. It was agreed that the details of the keywords be left to a sub-group.
The proposed format looks like this:
Extension
Primary header [0]
Spectrum [1]
Header
2D Float image
Statistical error [2]
Header
2D Float image
Data Quality [3]
Header
2D Short integer image
Table header
Position table [4]
Table header | Optional
Science table[1] [5] |
Table header |
Science table[2] [5] | Optional
.
.
It was decided that the position table of row, X,Y (extension 4) be mandatory. The further tables are optional but each should also contain columns for row, X, Y. The headers must contain the mandatory FITS keywords. Others that are specifically required can be added.
There was discussion about how to select apertures such as for star and sky when processing. This is facilitated by additional tables which can have each spectrum indicated as star or sky by a column whose meaning would be indicated in the header.
There was discussion about being able to accommodate spectra with different start wavelength and step. This is not possible under FITS so all the spectra must have the same start wavelength and step. If some spectra do not cover the full range, then the missing data can be handled by the data quality.
Martin advocated test data to exercise the format.
It was decided to set up a "task force" to produce a detailed definition of the format including keywords. This would be chaired by Markus (who is Work Package Coordinator in the RTN) with Pierre, Yannick and Martin as members. Their timeline is:
Software platform
There was heated discussion about what software frame we want to choose for
the Euro3D development. There are seen to be three possibilities:
C, IDL or IRAF. Midas is no longer a viable contender. As far as producing
software, C or IDL were seen as advantageous since IRAF runs from scripts
which are notorious to maintain (although the new PYRAF overcomes this
problem to a great extent). Martin expressed concern about the amount of
manpower we could realistically expect from the RTN. We cannot expect
post-doc astronomers to spend half a year producing a large C library
for example. The top level must provide a scripting environment, I/O
and display. Pipelines can obviously work without display. One suggestion
was that C routines with well defined calling parameters be used and so
the scripting would require little intelligence to run. A library of
C tasks could then be run from a script or from IRAF or IDL.
This is clearly the beginning of this debate but we must arrive at a decision at the next meeting well in time for the first post-docs of Euro3D to begin work within a framework and not spend many months on software development. It is clear that Euro3D should adopt a realistic but product oriented approach to its software products.
ESO Workshop on VLT 2nd Generation Instrumentation
Jeremy provided a brief overview of the talks which involved 3D spectroscopy.
There were nine talks (out of 54 total) which discussed 3D instruments or were
concerned with 3D spectroscopy. Given that a number of talks were more
general in nature then 3D spectroscopy clearly represents a bit slice of
future instrumentation on large telescopes. There was perceived to be a division
into two approaches:
Surveys of large fields collecting spectra of all objects
Instruments with mini-IFU's for spatially resolved spectra of many small targets
The former depended particularly on searching for targets with a one to a few
emission lines which could be missed by broad filter imaging. One example
was MEIFUS, a proposal for a million element IFU. See the
Jeremy provided a 5 minute presentation of the Opticon 3D WG at this workshop at the beginning of the 2nd day.
NEXT MEETING:
The next meeting of the OPTICON 3D Spectroscopy WG is not yet decided but
there is an offer to hold it in Leiden. A data following the RTN decision
is favoured, so around November-December.