FASTEX Central Archive Ship Quality Controls

But, for meteorological experiments analyses, and especially for case studies, a measurement can be very important in the interpretation. In these cases, the real time quality control processing appeared insufficient.

Following that, the FCA decided to improve the validation of the commercial ships surface measurements data set. This includes:

Here are the main problems found:

Data checking

Gross limit checks

As real time gross limits appeared too large for the FASTEX domain and period, the following values were used to complete the control processing. For each doubtful observation, data were visually examined and compared to neighbouring observations (in space and time). If a problem appeared, the wrong value was flagged and, in some cases, corrected (5: good, 6: maybe, 7: bad, 8: estimated); see the data correction section for more details.

In the table: B = bad, Q = questionable, E = estimated, G = good, P = pressure, T = temperature, RH = relative humidity, FF = wind speed

Parameter	Gross Limit Check real time values (FASTEX domain and period)	Gross Limit Check FCA values	Parameter flagged	Flag applied
Mean Sea Level Pressure	< 910 hPa or > 1080 hPa	< 960 hPa or > 1040 hPa	P	G/Q/B/E
Temperature	<-30 C or >50C if latitude<45N <-80C or >35C if latitude>45N	<-20C or > 30C	T/RH	G/Q/B/E
Relative Humidity	T$$D<-35C or >35C if latitude<45N T$$D<-85C or >30C if latitude>45N	< 30%	T/RH	G/Q/B/E
Wind Speed	> 60m/s if latitude<45N >50m/s if latitude>45N	> 35 m/s	FF	G/Q/B/E

Note that as the humidity checks went on, several ships appeared as having lower humidities than neighbouring ships, but without any way to settle if these data were doubtful or not. Flags were not modified, but low humidity values must be used with care .

WMO ship's call sign checks

A number of messages arrived from the GTS with a bad ship's call sign. There are several reasons for that: typing error when the message was sent on the GTS, problem of transmission, use of the name 'SHIP' or of the date instead of the call sign. These erroneous call signs were found by looking at too short or too long call signs (they usually have between 4 and 8 letters), by looking at ships which broadcasted less than 5 messages during the FASTEX period, by looking at call signs which were not referenced in the WMO ship list.

For each doubtful call sign, the message was compared to the other messages received in a spatio-temporal window of +/- 12 hours and +/- 3 degrees in latitude and longitude. The message was also compared to the messages sent by ships with closely related call signs. Some of these doubtful call sign were corrected in this way, without any uncertainty.

Following that, it appeared that a number of messages were in fact corrective messages, but not identified as it because the call sign was not correct. Only the good ones were keeped in the FCA.

The following table sums up the different cases:

nb of processed messages	cause	action
954 (0.55%)	unknown call sign	statu quo
451 (0.26%)	unknown call sign	corrected into a known call sign
2137 (1.2%)	unknown call sign	call sign='SHIP'
820 (0.47%)	duplicated transmissions	deleted

Ship trajectory consistency

In the real time quality control processing, a ship trajectory consistency already exists. It was completed in delayed time by the FCA in the following way.

Using parameters available in the SHIP messages, several checks were made on each ship trajectory:

locations of a ship were used to calculate, between every two successive locations, an average speed over the period between these two locations ( < 48 hours).

if the ship course speed was included in the message, and the difference between this average speed and the ship course speed was greater than 12 m/s, the location was checked.

when the ship course speed was not included in the message, every average speed greater than 50 km/h was checked.

the ship locations were also compared to the locations estimated using the previous location and the ship course heading and speed. When the distance between two locations was greater than a limit value, the ship locations, heading and speed were checked. This limit value depends on the ship speed, the time between two successive observations, the heading steadiness, and also includes the lack of precision due to the WMO coding itself, estimated in the following table:

parameter	precision in the SHIP message	lack of precision in km
latitude	0.1 degree	11 km at 50 degrees N
longitude	0.1 degree	7 km at 50 degrees N
ship heading	360 degrees / 8 sectors=45 degrees	speed ~ 15km/h, steady heading, 6h interval: ~48km, speed ~ 25km/h, steady heading, 6h interval: ~66km
ship speed	by 5 kt intervals

We can see that the error just due to the coding appears to be quite large. Nevertheless, this kind of control allowed to find several wrong ship positions. As an example, the following limits were used in this processing:

time between 2 messages	ship speed code	ship speed	limit value
6 hours	2	15km/h	90km
6 hours	3	25km/h	130km

Each doubtful location was visually checked, flagged if necessary and, in some cases, corrected (see the data correction section for more details).

Land/Sea mask

A new land/sea mask was used, more accurate than the operational one, calculated from the USGS Microwave databank II coastlines, lakes and islands at 0.1 degrees resolution, which is the precision of the ships location.

Positions flagged as being over land by the automatic QC procedure were checked with this new mask:

if 3 or more of the 8 points surrounding the position of the ship were water points, then the ship was automatically flagged as good;

for the other doubtful locations, values were visually checked and a geographical atlas was used in some cases. Flags were updated if necessary and, in some cases, data were corrected (see the data correction section for more details).

Controls on pressure

Foreword: in a SHIP message, the group '4PPPP' is used to report the air pressure at mean sea level. Most of ship messages include it. But another group exists to report the pressure at station level (group 3$P$₀P₀P₀P₀). Its use appeared rare in SHIP messages.

A first check was made concerning several ships reporting pressure using the 'group 3'. Two different kinds of ships appeared:

ships with only a few 'groups 3' (between 1 and 5 data): for these ships, it appeared that data included in the 'group 3' could not be pressure values. These completly aberrant values were removed.

ships with a lot of 'groups 3' (available in most of the messages from this ship). After controls (pression temporal evolution, comparison with 'pressure at mean sea level' of the same ship, comparison with mean sea level pressure maps), these data appeared consistent. These four ships/platforms are named 45135, 45139 (which are both in fact moored buoys reporting in SHIP message), DBFZ and FNXR.

Following checks were made on the 'pressure at mean sea level'. Afterwards in this section and to lighten the writing, the name of 'pressure' will be used for 'pressure reduced to mean sea level'.

Each pressure measurement was compared to 'neighbouring' observations (i.e. a 500 km square over a 12 hour interval, centered on the point which is controlled; this area might be extended when a few observations are available, or reduced for a great number of observations). When the difference between the pressure value and the average of the neighbouring observations was greater than a value set to 12hPa, data were checked with the use, in some cases, of the sea level pressure field from the ARPEGE analysis.

After that, for each ship, the temporal evolution of pressure was compared to limit values depending on the time between the two measurements; these limits have to include the fact that ships move between two measurements, and so pressure can increase or decrease faster than for a ground station.

temporal interval	limit on pressure tendency	limit on temperature tendency
1 hour	6 hPa	6 degrees C
3 hours	13 hPa	8 degrees C
6 hours	20 hPa	12 degrees C
12 hours	30 hPa	15 degrees C
24 hours	40 hPa	18 degrees C

A third check was also applied, using the pressure tendency. When, for a ship, the time between two pressure observations was equal to 3hours, and when the pressure tendency was available (allowing the computation of the first pressure using the second pressure and the tendency), the difference between the first value (from the message) and the calculated value was computed. If the difference was greater than 3hPa per 6 hours, pressure was visually checked. But tendency often appeared to be doubtful.

Controls on temperature

For each ship, the temporal evolution of the temperature was calculated and compared to the limit values given in the previous table. These limits include the fact that ships move between two measurements, and so temperature can increase or decrease faster than for a ground station.

Every time the temporal evolution was greater than the appropriate limit, data were visually checked, corrected if an obvious error appeared, and flagged according to the result of the checking.

Data correction

WMO ship's call sign

see section 2.b

Latitude and longitude data

In the following cases, and only in these cases and when the error is obvious, the decision was taken to correct position values:

error source	controls used to show up errors	parameters used to correct data
latitude/longitude coded in whole hPa instead of tenths of hPa error when coding the quadrant of the globe (used to determine latitude and longitude signs) latitude typed in place of longitude and vice versa in the message inverted figures or use of a neighbouring key on the keyboard	Ship trajectory consistency, see section 2.c for more details real time latitude and longitude flags set to 1 (suspect value) or 2 (ship over land, see section 2.d)	previous/next latitude/longitude values of the same ship (if not too far) use of the ship heading/speed when available




Pressure data

In the following cases, and only in these cases and when the error is obvious, the decision was taken to correct pressure values:

error source	controls used to show up errors	parameters used to correct data
pressure coded in whole hPa instead of tenths of hPa reading error on the instrument inverted figures or use of a neighbouring key on the keyboard wrong parameter in the pressure group in the SHIP message (problem of transmission?)	gross limit checks and/or comparison with neighbouring observations and/or temporal evolution of pressure for this ship and/or use of the ship pressure tendency and/or comparison with the closer sea surface pressure map (over Atlantic) and/or real time pressure flag set to 1 (suspect value) or 2 (bad value)	neighbouring pressures from other ships when available and previous/next pressure values of the same ship (if not too far) and pressure tendency when available




Temperature data

In the following cases, and only in these cases and when the error is obvious, the decision was taken to correct temperature values:

error source	controls used to show up errors	parameters used to correct data
temperature coded in whole degrees instead of tenths of degrees error when coding the sign reading error on the instrument inverted figures or use of a neighbouring key on the keyboard wrong parameter in the temperature group in the SHIP message (problem of transmission?)	gross limit checks temporal evolution of temperature for this ship and/or use of checks on humidity (a wrong humidity may be related to a wrong temperature) and/or real time temperature flag set to 1 (suspect value) or 2 (bad value)	neighbouring temperatures from other ships when available and previous/next temperature values of the same ship (if not too far)

Note that in some cases, it led to flag also humidity values (very low humidity values or obviously too high values due to an error in temperature for example).

Wind data

In the following cases, and only in these cases and when the error is obvious, the decision was taken to correct wind speed values:

error source	controls used to show up errors	parameters used to correct data
error when coding the unit of wind speed	gross limit checks real time wind speed flag set to 2 (bad value)	neighbouring wind speeds from other ships when available and previous/next wind speed values of the same ship (if not too far)

Flagging/correcting data

When a correction is made, the corresponding flag is set to 8 (corrected value).

When the correction is not obvious, but the value obviously suspect or bad, flags were used to improve the data qualification (set to 6 or 7). Conversely, some values flagged to suspect or bad by the real time control were obviously good data and the flag was set to 5 (case of a value set to suspect because of an error when decoding the message, or of a ship position over land, position which could be corrected).

Summary