In attendance:
Jeremy Allington-Smith j.r.allington-smith AT durham.ac.uk Roland Bacon bacon AT obs.univ-lyon1.fr Thomas Becker tbecker AT aip.de Veronique Cayatte veronique.cayatte AT obspm.fr Yannick Copin y.copin AT ipnl.in2p3.fr Pierre Ferruit pierre AT obs.univ-lyon1.fr Hector Flores Begona Garcia-Lorenzo bgarcia AT ing.iac.es Bianca Garilli bianca AT ifctr.mi.cnr.it Jorge Iglesias Markus Kissler-Patig mkissler AT eso.org Vincent le Brun vlebrun AT astrsp-mrs.fr Evencio Mediavilla emg AT ll.iac.es Jesus-Carlos Meunier Martin Roth mmroth AT ai.de Robert Sharp rgs AT ast.cam.ac.uk Juergen Schreiber schreib AT mpe.mpg.de Marco Scodeggio marcos AT ifctr.mi.cnr.it Lowell Tacconi-Garman ltacconi AT eso.org Jeremy Walsh jwalsh AT eso.org (Chair)
Jeremy Walsh welcomed everyone to the meeting. This was to be the last meeting of the 3D Spectroscopy Working Group established by the Optical and Infrared Coordination Network in Astronomy (OPTICON). The Working Group has had a short life, but a very successful one. The first three meetings were devoted to developing and preparing the RTN proposal which will provide a basis for widespread cooperation in 3D spectroscopy in Europe and dissemination of the technique. In preparing the RTN, two pillars were identified on which the RTN can build - a 3D data format and a software framework in which software will be developed and distributed. The 3D Format has been studied by a Task Force and presented in a draft document:- the details will be discussed so that it can be finalised for the beginning of the RTN. For the software it is important to reach agreement on the software environment prior to the RTN, in order to allow the young post-docs to work effectively from the beginning. The RTN then takes over the work of the 3D Spect. WG.
3D Data Format
Markus Kissler-Patig introduced the 3D Format draft document which had been
distributed prior to the meeting. The outline had
been provided at the Lyon meeting in June and discussed at the Milan meeting
in December.
Martin Roth asked whether the statistical error should be made compulsory. Several groups use statistical error arrays although propagation to derived quantities is infrequent. Jeremy W pointed out that there is a danger if statistical errors are compulsory and the arrays are filled with dummy values; some tasks may interpret the dummy values for processing. It was agreed unanimously that the statistical error image should be OPTIONAL. Keywords common to all the extensions should go in the primary header but could be repeated in each extension header. The Format document should specify the minimum keyword set to be held in the primary header (extension 0). In the Format document there were several instances of keyword names greater than 8 characters, which is not allowed in the FITS standard. Either the name can be shortened, or if this is not logically possible, then hierarchical keywords should be used. The number of Data Quality bits is completely used up in the present scheme allowing no flexibility for reduction flags; Martin suggested doubling the number of bits in the DQ images to 32 bit INTEGER. This was agreed and suggestions for further DQ flag values can be handled at a later time. It was asked if CTYPE is absolutely required for NAXIS=2 images. It was re-iterated that the first pixel in every image is (1,1) - this is the FITS convention.
There was an appeal for a good name for a spatial element! Offers welcome.
The Position Table engendered lengthy discussion. Martin showed a series of sample headers for images of various types from simple, to spatially binned to mosaiced. As far as the adaptive binned data is concerned, so far only Yannick Copin has used it extensively but the technique is clearly powerful (mostly for specifying the signal from background apertures) and should be handled by the Format. Martin showed example headers for mosaiced data (two IFU's). He pointed out the difficulties of combination of data with different atmospheric refraction; the "data cube" becomes a "data rhomboid". Evencio Mediavilla pointed out that the guide wavelength of the star used for tracking is also very important and that the refraction varies throughout an exposure. The airmass at the mid-exposure should be adopted. Martin added that to completely specify the atmospheric refraction it is necessary to specify the atmospheric temperature and pressure as well as the airmass, parallactic angle and wavelength. It was suggested that the atmospheric refraction data could go in the primary header.
Pierre Ferruit said that when making mosaics the WCS usually wasn't used (since often not trusted). There was discussion about storing several IFU data sets in one file. This would make a very complex structure. There could be a flag to specify if it was SIMPLE data (one IFU) or COMPLEX (more than one IFU, binning, etc). This would imply that the position table contains a history of each pixel (e.g. original pixels from several IFU's). Since the software will apply such operations as binning, the information and results could be put in the science table. On a point whether spatially overlapping pixels are allowed in the Format, Yannick appealed for the simplest possible position table. Making something that deals with all possible cases will lead to great complexity that will deter potential users. Martin suggested having a compound table for each binned pixel. There was a suggestion to use keywords for as much positional information as possible as there are fast tools to manipulate keywords. Rob Sharpe mentioned that CIRPASS data held the different pointing, IFU, etc in a different extension.
Martin showed an example of IFU data highlighting the problem of subtraction of emission line background (faint [S II] emission from a PN in a spiral galaxy). The background region could be an annulus and Martin asked how would this be stored in the format. It was stated that if a compound pixel was produced the data would be written as a separate file. A great deal of the complexity for compound pixels comes in visualising them. No unique centre can be specified and a centre could be coincident with an unbinned pixel. It was agreed that a compound table was required to specify the parameters of binned data The compound table would be an additional optional table. Unbinned data would not require this table. The row number of the spectrum would be the fundamental pointer. There was discussion about whether to put most of the information in this compound (binning) table in keywords or in the table. The GMOS experience was not to recommend hierarchical keywords; FLAMES uses many (non-hierarchical) keywords.
Thomas Becker asked about handling of polygonal spatial elements. Round,
square and hexagonal elements were explicitly treated by the Format. Currently there
are no instruments with polygonal (no. sides >6) in use. It was agreed that
the compound table should point to another table where the spatial element
shape is specified. For the simplest case of one IFU, no binning, there
would be no compound binning table and a single entry in the shape table.
It was agreed to make the shape table mandatory. There would be one entry
for each group (viz. IFU).
Pierre said that mosaicing shouldn't carry implications for the WCS; in
other words mosaiced data can be considered as produced by one
(meta-)instrument. Data sets
only hold the result of a merge between different IFU's not the separate
elements. This should be clearly stated in the Format document. It was agreed
that only a single WCS should be specified for each data set. Thomas stated
that he wanted to transform the WCS rather than the X,Y's of the elements.
For two files to mosaiced, either there are:
2 WCS's and two separate (X,Y) origins;
one set of WCS and two distinct sets of X,Y coordinates.
The SPEC_ID (now a string) could be used to distinguish different mosaiced
sets. Only mosaicing should alter the X,Y, not the WCS. Martin pointed out
that the WCS should refer to a specified wavelength in order to allow
correction for atmospheric refraction.
The Position Table now looks thus:
Row SPEC_ID X Y Npix GroupFor compound pixels (Npix>1), the X,Y has no meaning nor has Group. The optional compound binning table looks thus:
Row SPEC_ID X Y Group FlagThe row number is the same as the row number of the Position Table,and would thus be repeated as many times as there are pixels in the compound pixel. The Group refers to the entry in the Shape Table which specifies the shape of the (unbinned) IFU spatial element. The Shape Table looks thus:
Group Shape Param1 Param2 ...The parameters are defined by header keywords and specify pixel shape and orientation - e.g. SIZ1, SIZ2, angle, etc.
The Data Format document should be amended as per the discussion and distributed. Final comments should be collected with the aim of release by 01 June 2002. Martin undertook to take some typical IFU data and put it into the Euro3D format. Bryan Miller may also be interested in converting GMOS data to the Euro3D format and Martin (?) will contact. Bryan is trying to get the IRAF group to accept a 3D format, and it will hopefully be close to Euro3D, if not identical. Bianca said she would provide a translation for VIMOS data to Euro3D. When the Format is finalised, sample data in Euro3D should be passed to ESO DMD for comments.
Software platform
Lyon libraries
The discussions opened with Pierre presenting the Lyon C libraries. The
Lyon format is similar to the Euro3D one in having a file for the
spectra and associated FITS table for position and derived (science) data,
but they are separate files. Data and variance is stored. The library is
written in ANSI C and handles images, cubes and tables as basic structures.
There is a dedicated I/O library. There are also libraries of general
purpose tools and data reduction software. Tcl/Tk is used for the graphical user
interface and widget building to handle the formatting of command lines;
experienced users typically develop their own scripts and call the executables
directly.
C was selected for the I/O library since it is free, portable, has many numerical libraries available and access to higher level tools. It encourages good programming style. The image and table are loaded together and handled as structures. The I/O library is isolated from the applications which tend to be more complex. Changes to the I/O library are well controlled at Lyon whereas the applications are developed widely. The general purpose library covers session handling, argument parsing and error handling. All routines have a homogeneous frame and associated help text which can be manipulated by e.g. docxgen. /configure is used to ensure portability. Yannick, who regards himself as a non-programmer, said he very quickly got started with the packages and found it easy to program.
The choice of GUI was made before Python was widely available. MIDAS was used previously. IRAF proved to be too complex. Tcl and Tk were chosen (Python uses Tk) and visualization tools existed.
For the (XOasis) applications to use the Euro3D format, only the I/O library would need to be changed. The structure could handle the options in the Euro3D format. It was estimated that a few weeks would be required to determine how the Euro3D format would fit into the current C structures. Then the I/O library would need to be ported. Arlette Pecontal is the chief programmer. It was estimated that the software effort could realistically be 4 weeks. It was not clear if such a change to the I/O library would have repurcussions for the applications. Markus said that there was a possibility for ESO to provide some C programming support.
Pierre said that XOasis to be installed on the WHT would probably use the Euro3D format so that Lyon would want to convert the I/O library. Currently XOasis binaries are available publically but source code is not. A question was asked if the source code would be made available.
IDL
IDL is unsurpassed for rapid prototyping and development. The software is
open, well supported, feasible to support for applications and there have
been several instances of data reduction tools written in IDL. The vector
oriented approach is well suited to spectra. IDL can be linked with external C
code. Whilst IDL is fast (Martin
showed display of a 6144x4608pixel image using the ATV tool - similar to Saoimage),
its speed depends on the memory of the machine for large data volumes. Martin
said he considered IDL well suited to the RTN where the post-docs could not be
expected to spend more than 11 months of their two years on software
development. Expert programmers should not be expected or required for
the RTN post-doc positions.
The primary IDL astronomy applications library is Astrolib. This was originally written for the UIT satellite but has been greatly extended and includes many applications for general manipulations, astrometry, photometry, FITS image and table manipulation, etc. The library is no longer financially supported but is maintained and news and updates are edited by Wayne Landsman. There are also other IDL libraries available - IDLWAVE which can run IDL from EMACS, IDL on the WEB (simulations), Chandra software, ISO and NICMOS reductions, etc.
Thomas Becker showed a demo he had prepared reading simulated Euro3D format data. The header was modified and the position table transformed. The WCS transformation was made using Astrolib routines. Spectral selection over a region was demonstrated - the powerful function "where" was extensively used - and flux v. lambda plot made of the summed spectrum; this with 57 lines of IDL. A demo of the effect of the spectral distortion resulting from atmospheric refraction in extracting adjacent pixels was shown. A movie of GMOS data was shown and the interactive widget driven tool, developed by Thomas, for displaying images, extracting and displaying spectra was demo'ed. Bianca said she was impressed by this tool. A question was asked if IDL requires an I/O library and if someone could design an API (Application Programming Interface) for outside programmers.
In the ensuing discussion it was emphasized that IDL is a commercial product and that money would have to be found for IDL licences. Martin said that multiple Linux IDL licenses can be purchased rather cheaply. On a show of hands most said they had access to IDL, although sometimes the number of licenses was limited. Milan had no IDL. Martin emphasized that for an RTN there is no deliverable item, such as software (training is the "deliverable"). Jeremy W. reported that he had talked to Peter Quinn the ESO DMD head about possible software support by ESO after the life of the RTN. Peter said that ESO no longer is involved in data analysis software, so officially wouldn't support such tools. However DMD would be very interested to follow our developments and to use code whenever possible for the reduction of 3D data (this is a fuzzy boundary since even target acquisition with 3D data involves manipulation of the data). DMD would feel comfortable taking on any C code but not IDL.
Jeremy A-S emphasized that whilst software was important, the development of complex algorithms was to an extent independent of the software platform. He presented several Starlink documents written by Alistair Allen which collected together and listed available software for IFU and 3D applications and provided a cookbook. The RTN should develop research level software that gets to the results rapidly. The effort to develop user friendly, highly robust software is of another order and involves different skills. Distributing cookbooks to distill experience and use of software was favoured by several. It was emphasized that C code is definately not as compact as IDL and more difficult to read.
VIMOS software
Bianca showed some of the VIMOS commissioning data for the IFU and Jeremy W showed
a (highly preliminary) derived H-alpha image of the PN NGC 3918. The VIMOS
IFU data structure was described which consists of a FITS image and a table.
The VIMOS library puts the spectra and position table information in a structure.
The library also incorporates memory allocation routines, arithmetic, filtering,
and some mathematical functions taken from other packages (e.g. eclipse). Data
cubes can be built and spectra for specified positions extracted. The prototyping
for this code was done in IDL but was written in C for ESO requirements.
FLAMES software
Veronique Cayatte briefly described the FLAMES reduction software. Python is
used for the pipeline extraction of the fibre spectra but there is no specific 3D
library. eclipse with a Python wrapper is used for some applications. Parts of
the pipeline can be used stand-alone. The Python code will be converted to
C for ESO DMD support. An Ancilliary Data Analysis Software
(ADAS) is under development for e.g. cross-correllation, narrow band filter
colours, etc; this is wholly in Python. Markus agreed to inform the RTN of
the IFU applicable FLAMES software and eclipse routines which are ESO
supported.
Integral IDL package
Begona described the Integral Data Analysis (IDA) IDL package. This was
written by a student at IAC in 2 months. It is windows based and uses the
IDL Interactive Display tool of Aaron Barth. FITS or IRAF multi-spectrum
files and ASCII tables for positional information are the typical input.
Demonstration of the very adaptable display - overplotting of spectra on
maps, kinematic maps, ionization maps, etc was shown. The package was
modular and used the Astrolib IDL library.
eclipse
Markus briefly decribed the ESO eclipse library. This was first developed for
ADONIS then for ISAAC, and will support VIMOS, CONICA, SPIFFI, etc in the
future. It consists of a C library and the functions can be embedded
in other C routines. For 3D data, a cube structure is defined. Contributed
functions can also be included and it will be supported by ESO.
National VIMOS centre
Vincent briefly described the setting up of a national software centre for
spectral analysis, primarily for VIMOS but also including FUSE. This is
available to the French community and will facilitate VIMOS data mining in
the VIMOS survey - 3x10^6 targets and 15000 spectra. The server is
located in Marseille and supports internet access and visitors. The software
in the centre is mixed - C, Perl, shell scripts, and Fortran.
Concluding discussion
I try to summarize the many-sided discussion:
Resolutions
Preparations for the RTN
Martin showed some slides of the
CORDIS Web pages
relating to contract negociation.
In fact there did not really appear to be any question of renegociation. The
question of relocation allowance which was raised cannot be considered. The
contract preparation documents had been submitted. The current scheme is for
start of the RTN on July 1. Payments can only flow after the membership agreements
have been signed with each node. Delay in payments are expected by the EU.
First financial distributions can be expected around September.
Official signatures on the membership agreements are required and vigilance about
any delays over the summer should be observed. For some nodes it is not
allowed to advertise for positions before the money has arrived. Annual reports,
coordinated by Martin are critical for keeping the money flowing. The final 15%
of the payment is paid after the final report!
There is a possibility to include scientists not named on the original proposal but this has to be properly handled. Changes in teams must be quickly communicated. If a team leader moves, then the membership agreement must be renegociated.
Employment conditions
The age limit of 35 is strict but terms of military or civil service or
child care can be taken into account. US citizens can be RTN post-docs but
only if they have spent the last 5 years in Europe. Associated states
count as European countries. Reference to the
CORDIS RTN pages
should be made for clarification. Since applicants are locally employed by the
team institute they must fulfill local contract conditions.
Advertising
Jeremy showed a straw general poster advertising the RTN, and comments
were incorporated.
This is available.
This will be put on the Euro3D Web pages. Each node can then modify this
to provide a local profile, outlining the local interest and expertise.
These are to be sent to Martin by 15 April. The advertisements
can also be posted on the CORDIS Web pages. Each team can advertise widely.
An AAS general advertisement, based on the above will be prepared with a view
to inclusion in May and June.
It was agreed that a colour poster would be good publicity. Please send jpegs to Martin of recent glamorous images for possible inclusion. Markus will ask the ESO graphic artist Ed Janssen to make up a poster from the collected images. Jeremy W will write an ESO Messenger article describing the RTN.
Kick-off meeting
The RTN begins on 1st July and it was agreed that July 1-3 would be well
suited to the Kick-off meeting. This will be held in Teneriffe at the IAC in
Santa Cruz. The meeting will be a mix of science and instrument presentations,
guidelines for software and
organizational preparations. It was agreed that an EC representative was not
invited.