In attendance:
Thomas Becker tbecker AT aip.de Petra Boehm pboehm AT aip.de Veronique Cayatte veronique.cayatte AT obspm.fr Pierre Ferruit pierre AT obs.univ-lyon1.fr Bianca Garilli bianca AT ifctr.mi.cnr.it Markus Kissler-Patig mkissler AT eso.org Dario Maccagni dario AT ifctr.mi.cnr.it Juergen Schmoll jurgen.schmoll AT durham.ac.uk Marco Scodeggio marcos AT ifctr.mi.cnr.it Lowell Tacconi-Garman ltacconi AT eso.org Jeremy Walsh jwalsh AT eso.org (Chair)
Jeremy Walsh opened the meeting. The first phase of the work of the Group in successfully receiving RTN funding for Euro3D has been completed. There are however several important groundstones to be laid for the smooth running of the RTN - namely the 3D Format and the platform for the software to be written and disseminated. This coordination activity requires that the 3D WG should continue until funding for the RTN flows, at which time OPTICON may review the role of the Working Group.
Since Pierre was delayed, and he was one of the members of the 3D Format "Task Force", it was decided to discuss steps towards the RTN first.
Realising the RTN
Petra Boehm distributed a photocopy of the letter reporting the evaluation
of the RTN submission. Jeremy commented that the high scores on Quality of
Network partnership and training programme helped bring the proposal
well above the cut-off score (considered to be ~81). However the comments
in the letter were quite critical and did not seem to equate to the high scores.
There was mystery why we received such high scores for the training aspects,
especially relevant to industry. The letter of invitation to negotiate for
an RTN contract is expected to be recieved in the period from mid-December to
mid-January. Petra distributed copies of the Forms A5.1-A5.3 which must be
filled out by each partner. These forms are more detailed than those in the
RTN application and require administrative input so sufficient time must be
allowed. Failure to return such documents in time could endanger the future
of the RTN. Submission of these forms would then be followed by negotiations
on the contract. Dario Maccagni reported on another RTN in which he had been
involved and said that money did not start to flow until a few months after
contract signing. A realistic estimate of when the Kick-off meeting could
occur is probably June. Dario Maccagni suggested that some science could be
included in the Kick-off meeting, as done for other RTN's.
There was discussion about communication within the Network. Petra was task coordinator for a Newsletter. There was discussion about having a pure email Newsletter or one available on the Web. Markus Kissler-Patig said that he found FRAMES awkward to use and it was not now very frequent among Web pages. It was agreed that the most logical procedure is for each Task Coordinator to generate a short report which is then sent (preferably in ASCII) to the Work Package Managers who will synthesize. These reports are then sent to Petra to distribute. The Web would be used, with an email notification of new reports. The reports on the Web should be in PDF or postscript so they could be printed and read; there would be no printed version circulated. There should be such reports 4 times per year and they would be in a password protected section of the Euro3D web pages. Every 6 months a public portion of these reports should be creamed off and put on the Web pages for access outside the Network. This would contain highlights and important news - e.g. release of a new analysis tool, hot science result. The first newsletter should be sent out before the Kick-off meeting.
Bianca agreed to prepare four mailing lists - one for each of the three Work Packages and a general one including everyone. These would reside somewhere for email distribution, in order to avoid long lists of names in email headers which can be read by malicious robots.
The search for appropriate post-docs for the RTN was discussed. The
rules for RTN post-docs are that:
she/he must be a national of an EU Member state or of an Associated State
or have resided in the EU at least 5 years prior to appointment. She/he
MUST NOT be a national of the state in which the participants research team
is located and must not have carried out normal activities in that state
for more than 12 of the 24 months prior to appointment. She/he must be under
35 years of age.
The fact that the post-docs sought must be able, and prepared, to produce software
may mean that we loose some totally research-oriented individuals, but this
is the price to be paid for the Euro3D project (and one of the reasons we got
the RTN in the first place). The software experience will put them in excellent
positions in the wider astronomical world. Most good quality
post-docs write their own software anyway.
It was agreed that a general notice should be prepared, for submission to the AAS jobs bulletin, about the time of signature of the contract, advertising the RTN, the partner institutes and giving advance notice that we will be looking for post-doc positions. This notice in email form could also be sent by email to many European institutes. jeremy will prpeare a draft. More specific notices will be issued later. Timing was seen to be quite critical if we are not to completely miss the round of post-docs seeking positions from September 2002. Fresh post-docs from Italian institutes come onto the job market around January. Given that the RTN is for three years and should start around June, then two year post-docs starting as late as June 2003, or perhaps a few months later, are possible.
Markus Kissler-Patig reported that there were a number of candidates for ESO Fellowships who expressed interest in working with 3D data and techniques. Markus will contact the ones who were not successful in gaining an ESO fellowship, informing them about the RTN and employment possibilities.
Already there has been some movement of personnel between institutes (ESO have accreted Lowell Tacconi-Garman from MPE and Rachel Johnson from IoA). These groups should make sure to identify a science contact to ensure smooth and speedy communication with the other teams.
A booking for an IAC Winter School around Dec 2003-Jan 2004 should be made.
3D Data Format
Markus handed out the preliminary draft of the 3D Format prepared by the
Task Force (Yannick Copin, Pierre Ferruit, Martin Roth and Markus). Martin
had sent his input to the task force to the whole of the 3D Spect WG.
It was agreed that the Task Force will produce a single document describing
the Format. Markus will use a Latex template and propose a general template
for all subsequent Euro3D documents.
As agreed at the Lyon meeting there will be a Primary Header - Extension 0. The spectrum will be compulsory (extension 1) as will the Data Quality (extension 3). The DQ must be compulsory to flag spectrum points without data (since all spectra may not have identical wavelength starting value). A Euro3D keyword was suggested. This could have the meaning that the FITS file contained all required structure and extensions. However since these must anyway be checked for, the Euro3D keyword could just be a label. Whilst EXTVER and EXTLEVEL are not necessary, they should be used in case there was evolution of the format and new versions were created.
It was agreed that the image and statistical error be in FLOAT (BITPIX -32) and the Data Quality in BITPIX 16. The only header keywords that need to be Double are coordinate and WCS keywords.
There was discussion about wavelength units. They don't have to be prior defined and the CTYPE and CUNITS keywords can specify. IR data may favour microns over nm or Angstroms for optical data. A mix of units between the spectrum and the statistical error would not be supported. Coordinates must be in decimal degrees.
Position table
There was a very lengthy discussion on this topic, which is extension 4.
This aspect is central to the adopted format, given that spectra are stored and the
association with position on the sky is mandatory to produce maps. The
most stringent driver is the ability to handle adaptive pixel sampling
(i.e. increasing the signal-to-noise of a spatial region by adding spectra
from neighbouring (spatial) pixels). It was agreed that if the pixel shape is simple
(square, round or hexagonal) then this should be indicated in the header of
this extension without resort to extra columns in the table. Polygonal
pixels can be also specified; the definition of the corners would appear in
the header, either through a vector of X and Y offsets or a vector
of lengths and angles. To handle
pixels which are built up from the basic IFU pixel units, a keyword SPECIAL
would be used and the exact specification of the pixel boundary would be
contained in the table.
The table would look so:
Index Ref.No. Delta_X Delta_Y Flag
The reference no. would always refer to the row of the spectrum in
the image. Delta_X and Delta_Y would be specified in instrument coordinates.
This implies that the extension header must have a WCS to handle conversion
to sky coordinates. The upcoming version of the CD matrix for the WCS
allows parameterization of complex distortion, not available under the
previous CD matrix (see Greisen, E. W.,
Calabretta, M. R., 2002. "Representation of world coordinates in FITS",
to be submitted to A&A;
Eric Greisen's FITS
page).
The Flag column can include
extra information such as the origin of the spectrum (sky, different IFU
or mosaiced; the Task Force will propose a detailed convention). There
was a suggestion to put in a keyword to state whether
the data had been corrected for differential atmospheric refraction or not.
The reduction steps are recorded in the HISTORY keywords. For data
not corrected for atmospheric refraction, a keyword specifying the
pivot wavelength at which the WCS applies should be added.
In the Task force document on the 3D Format, the suggestion is made to specify the boundary of a compound pixel by a vector of lengths and angles. This is adequate for closed regions but not for non-contiguous regions (e.g. sky spectrum composed from several regions). The first angle (offset from the centre of gravity of the compound pixel) should be absolute but the others can be relative to the last direction. Absolute angles may be safer to avoid rounding errors propagating. The compound pixels have a calculated pixel centre. The visualization software will have to be able to reconstruct the area of the compound pixels in terms of IFU pixels. The case of FLAMES where there can be multiple IFU's (sky IFU's) with different shape and orientation to the large IFU was considered in these arguments. To allow non-contiguous regions with the same spectrum, it would be necessary to have another set of table columns which listed the number of included pixels and then their X and Y positions. These could be a vector in the table as for the offsets and angles and signalled by a dedicated keyword. The Task Force will propose a format where both would be allowed.
Science table
It was decided that the spectrum ID should be the basis of the derived quantities
in the science table. The position information should come from the position table.
Task Force report
The Task Force will deliver their report by 01 February. This will be circulated to
the 3d Spect. WG members. Following input and email discussion the report will be
circulated to FITS guru's (Preben Grosbol, Eric Greisen, Francisco Valdes)
for comment before official adoption (target 01 March 2002).
ESO Call for 2nd Generation Instrumentation
Jeremy advertised the ESO call for proposals for 2nd generation VLT instruments. Following the Workshop on "Science Drivers for future VLT/VLTI Instrumentation", ESO is inviting preliminary proposals for four instrument concepts. Of most interest to the Working Group is one for "a wide-field 3D spectrometer (3D deep-field surveyor)". See for more information. It clearly shows that 3D spectrometry is becoming a forefront technique and that the success of the RTN is going to be important for the exploitation of the vast amount of data that will be taken.
Software platform
Since the last meeting in Lyon there has been a significant development.
The Lyon group have offered their C library to the 3D WG and to the RTN
to be the basis of the software platform. The group have an extensive
history of work in 3D spectroscopy and their library is the most
developed one available. Arlette Pecontal maintains the software and
it is well structured and documented. Routines have documentation
which can be extracted. The Lyon format (for Oasis and
SAURON) is different to the proposed 3D Format so that coding is required
to interface the Format to the structures of the library. However this
is estimated at <1 week of work using the cfitsio library. All the basic design work
of developing the structures for the data has already been done. Some
modification resultant on the differing format may be required, but this
is considered to be minor. Pierre estimated that the time of an expert
needed to
port the library can be split into two steps: 1 week for the definition of
the structures and routines; ~1 month for the porting and testing of the
code. With this offer, the RTN would be commencing
with a very mature library which would form a sound basis of the
application development planned for the RTN. Putting the library routines
together into applications could be done with a scripting language such
as Python, although Unix scripts which call executables could be made.
Pierre Ferruit briefly described the Lyon software:
---------------
| EURO3D Format |
---------------
|
| I/O [cfitsio or qfits*]
|
--------------
| C structures |
--------------
|
|
----------------------
| Access to structures |
| eg. handling | ------------------
| selections |----| Generic tools |
---------------------- | for manipulation |
------------------
|
------------
| User tools |
------------
* qfits is the version of cfitsio used by eclipse (now renamed CPL - Common
Pipeline Language)
The ESO RTD is used for image and graphical display and the user interface
is developed under Tcl/Tk. The GUI based on Tcl/Tk is used to set up common
reduction procedures and command lines which consist of an executable with
required parameter values, with at least the mandatory ones filled. Expert users
often edit command lines directly. Pierre gave a short demo of
XOasis.
Pierre agreed to provide, before the next meeting, an inventory of the routines
in the library.
Based on experience, Pierre emphasized that design of 3D software, and in particular the data structures, was critical. Given the choice of the 3D Format, it is necessary to develop code which allows simple and efficient access to the data based on positional information, since the access will not instantly be to array coordinates. Pierre will provide an inventory of the Lyon C routines before the next meeting.
A C library is in principle platform independent and is backwards compatible. IDL is also platform independent but there can be problems with new versions. The decision for a C library is in principle independent of the scripting language (e.g. Lyon uses Tcl/Tk). Python is available for many platforms and in addition with the development of Pyraf allows command line access to all of IRAF. All this software is free. IDL costs real money, the argument being that it is cheaper since there is so much software available already. However the RTN would have to distribute a large number of licences for outside users who didn't already have IDL and also some development licences, which are expensive.
Thomas Becker said that most of the IDL IFU software so far developed at Potsdam was for data reduction. For data analysis, IDL has the advantage that standard routines and code from the NASA/GSFC Astrolib library are available for many tasks. It became clear that there would still need to be intermediate routines (equivalent of C structures) developed if standard IDL routines were to access 3D data via the position table information (i.e. generate array indices). Potsdam will produce an inventory of the IDL routines for 3D spectroscopy.
There was discussion about control of software which would be put in the library by the developers in the RTN. Some thought needs to be given to Version Control, and there are tools to handle this. Pierre mentioned the experience of the SAURON project where several groups had access to the core library and developed their own applications. There will have to be a controller of the library, who maintains the library and is responsible for running tests to ensure that new additions are compatible with existing routines and that bug fixes in one routine do not break others. Test data and routines must be set up; Pierre said that the Lyon group already have such a procedure. Markus said that ESO would most likely be the final repository of the software delivered during the RTN; it is thus important that the platform be in line with ESO's support policy.
There was brief discussion about a mini-workshop on software standards for coding and documentation. It was suggested that this be part of the RTN Kick-off meeting. The 3D Format and adopted platform would be described.
It was agreed that there had been plenty of discussion about the platform and that
if the young researchers are to be able to start development when they join the
RTN (in practice probably September 2002 at the earliest), then the Format and
core routines MUST be ready. The FINAL decision MUST take place at the next meeting;
allowing six months to finalise.
NEXT MEETING:
The next meeting of the OPTICON 3D Spectroscopy WG is not yet decided but
it will be held in Marseille or Teneriffe. A date before Easter (end March)
is favoured.