6. Evaluation and Recommendations

• 6.1 Survey-Based Datasets
  • Measurement Process
  • Sampling Process
  • Inference Process
  • Dissemination
  • Survey Management
  • 6.1.1 Commercial Accommodation Monitor
    • Evaluation
  • 6.1.2 Domestic Travel Survey
    • Evaluation
  • 6.1.3 International Visitor Arrivals
    • Evaluation
  • 6.1.4 International Visitor Survey
    • Evaluation
• 6.2 Aggregation-Based Datasets
  • 6.2.1 Forecasts
    • International Visitor Arrivals to New Zealand 2002-2008
    • International Length of Stay and Expenditure Forecasts 2002-2008
    • International Tourism in New Zealand - A Regional Perspective 2002-2008
    • New Zealand Outbound and Domestic Tourism Forecasts 2002-2008
    • Regional Forecasts
  • 6.2.2 Tourism Satellite Account (TSA)
    • Adequacy of Data Sources
• 6.3 User Needs Assessment
  • Support and Training
  • Roadshows and Demonstrations
  • Training Courses and Workshops
  • Guidelines for Appropriate Use
  • Providing Technical Support
• 6.4 Data Management Systems
  • 6.4.1 Supplementary Data
    • Supplementary Surveys
    • Administrative Data
    • Travel Price Index
    • Data Modelling
    • Linking Research
  • 6.4.2 Classifications
    • What Is a Classification?
    • Why Have Classifications?
    • Accommodation Type Classification
    • Activities Classification
    • Age Classification
    • Country Classification
    • Local Government Areas Classification
    • Main Reason for Travel Classification
    • New Zealand Destinations
    • Personal Income Classification
    • Reason for Not Travelling
    • Regional Tourism Organisation Classification
    • Transport Type Classification
    • Travel Companions Classification
  • 6.4.3 Data Documentation
  • 6.4.4 Data Ownership
  • 6.4.5 Data Security
  • 6.4.6 Dissemination
    • Data Publishing
    • Microdata Access
    • Regional Estimates
    • Regular Reports
  • 6.4.7 Change Management
  • 6.4.8 Risk Management / Impact Analysis
  • 6.4.9 Regular Review

6.1 Survey-Based Datasets

Evaluation of the four survey-based datasets consisted of assessing the survey methodology, survey process, and dissemination.

The objectives of evaluating the surveys were to:

• ensure the survey design is sound, and
• ensure the resulting statistics are of adequate quality.

A sound statistical survey involves the collection of information from some or all units of a population using a frame and well-designed concepts, methods and procedures based on statistical theory for selection and estimation. This allows measure of the levels of bias, variability and cost so that a survey design can be developed to achieve the stated objectives. (Statistics New Zealand, 1998)

The elements evaluated have been grouped under five headings:

• Measurement Process: getting the information (the collection method used)
• Sampling Process: getting to the right people (the sampling frame used, the sample selection method used, the size of the achieved sample)
• Inference Process: getting the big picture (weighting the sample up to the total population)
• Dissemination: getting the information to users
• Survey Management: conducting the survey

Measurement Process

Survey Objective: does the survey objective state the need for the survey, the information being collected and the key outputs?

The survey objective is a detailed statement about the need for the survey, what information is being collected, and the key outputs. It distinguishes the small number of key pieces of information, which are essential, from the less important things. A clearly defined objective justifies the survey and provides a basis for assessing amendments and measuring impacts of changes. The survey objective should:

• state why the data is needed and what it is used for;
• state the periodicity of the data (how often the data will be released), where applicable;
• be based on the main uses of the statistics;
• be expressed in terms of key variables, and the main classificatory variables, along with standards for accuracy, timeliness and presentation;
• be followed by a description of the target population of the survey, where applicable;
• use terms consistent with WTO definitions.

Questionnaire: do all the questions contribute to meeting the survey objectives (and are therefore considered necessary)? Is the questionnaire structure adequate? Are there questions on satisfaction and motivation?

Data Capture: is the interview process adequate?

Good interviewers are critical to the success of the DTS and IVS. They must have appropriate training and tools. Factors such as training, experience, interpersonal skills, rapport building skills and staff turnover will have an impact on the response rate of the survey.

Please note that the CAPI (computer assisted personal interviewing) and CATI (computer assisted telephone interviewing) programs have not been evaluated. The evaluation of the questionnaires is based on hard copy versions. The design and wording of the individual questions were not assessed.

Respondent Burden: is the respondent burden minimal?

Classifications and Definitions

Sampling Process

Target population / Sample population: are these well defined?

Sampling Frame: does the survey frame conform with the target population and minimise undercoverage and overcoverage (including duplication)?

Sample Allocation: is the sample allocation adequate?

Sample Size: is the sample size adequate?

Sources of Error: have possible sources of error been identified?

Non-response: what effect does non-response have on the survey? Is it monitored?

Inference Process

Weighting: is the weighting process adequate?

Imputation: what imputation procedures are used? Is this adequate?

Expenditure estimation: are the estimation methods sound?

Outlier Treatment: is the method for treating outliers sound?

Dissemination

Quarterly Estimates: what reliable quarterly data is available?

Regional Estimates: what reliable regional data is available?

Metadata: what metadata is currently available?

Accessibility: is the information readily accessible? Accessibility will be affected by the implementation of Harmoni on the Tourism Research Council New Zealand website, and dissemination of data via the new website.

Survey Management

Quality and Performance Measures: are the quality and performance measures adequate?

The Ministry of Tourism is now responsible for the purchase of the core tourism datasets; in this role the Ministry is also responsible for monitoring the quality of the data and the performance of the providers.

To assess the quality and performance, the Ministry needs to:

• establish quality and performance measures for each dataset
• receive regular reports from the providers
• regularly audit the data.

Important quality measures include response rates, follow-up rates and counts of non-response by reason. When these measures are available at any level at which estimates are produced at various stages of the process, they can serve both as performance measures and measures of data quality. (Statistics Canada, 1998)

In addition, each aspect of the survey production cycle should be audited at least once a year. The purpose of the audit is to verify that operational procedures and controls comply with the documented procedures and to determine their effectiveness in delivering products and services that are fit for purpose. (Holt & Jones, 1998)

Documentation of procedures: is there adequate documentation of data processing, imputation, weighting, estimation and outlier treatment procedures?

Consistency with other datasets: is the dataset consistent in concept and definition with the other datasets?

Timeliness: is the data available in time for it to be useful?

Frequency: is the data released often enough?

Stakeholder group: is a stakeholder group necessary, and has one been established?
Stakeholder groups play a significant role in providing advice and feedback on survey and data issues, and can also help monitor changes in technology and new developments, assist in assessing the impact of changes to surveys or questionnaires, and. The stakeholder group should consist of representatives from the Ministry of Tourism and Statistics New Zealand, and key user representatives.

6.1.1 Commercial Accommodation Monitor

Information for the Commercial Accommodation Monitor (CAM) is obtained from a survey conducted by Statistics New Zealand. This survey provides monthly data on capacity, occupancy rates and guest nights for each month, plus employment and guest nights by usual region of residence of guests for every third month.

The purpose of the survey is to provide information on available capacity, occupancy rates and employment for the accommodation industry.

The target population for the survey is all establishments that are classified as short-term (less than one month) commercial accommodation providers operating in New Zealand. The survey frame consists of all geographic units (establishments) on Statistics New Zealand's Business Frame with the Australian and New Zealand Standard Industrial Classification (ANZSIC) codes of H57100 (Accommodation) and part of H57200 (Pubs, Taverns and Bars). These geographic units must belong to an economically significant enterprise (that is, an enterprise that is GST registered, with a turnover of at least $30,000 per annum).

The survey is a census conducted on a monthly basis by using questionnaires that are completed by the respondent and mailed back to Statistics New Zealand. There are two questionnaires:

1. Short Form - this questionnaire is sent out in two of every three months, and collects information on the number of rooms or sites sold, total guests and guest arrivals.
2. Long Form - this questionnaire is sent out every third month, and collects additional data on the origin of guests and number of full-time and part-time employees, as well as the information collected on the short form.

The survey began in July 1996.

The CAM report is produced for Regional Tourism Organisations (RTOs) by Statistics New Zealand. Each report is written for a specific RTO area, with sub-area information provided for some larger RTO areas. The report also includes comparisons between the RTO area and national figures.

Evaluation

(1) Survey Objective

The current survey purpose is stated as:

To provide information on available capacity, occupancy rates, turnover and employment for the Accommodation Industry.

This needs to be revised:

• as turnover data is no longer collected
• because survey undercoverage means that data is not available for the whole accommodation industry
• to include the frequency of the survey (monthly)
• to include the main uses of the data.

The survey objective should be discussed by the stakeholder group (Recommendation 11) before implementing.

(2) Questionnaire

The survey is conducted using two forms, a short form and a long form. The short form captures information on capacity, occupancy rates and guest nights. The long form captures additional information on origin of guest and employment data. The options for origin of guest (for international visitors) are Australia, Japan, Korea, Other Asian country, UK or Ireland, Germany, Other European country, North America, Other Country, and Don't know.

It is unclear how this list was derived and it does not reflect the key markets identified by Tourism New Zealand. It is also unclear whether "Korea" includes both North Korea and South Korea, and why USA and Canada are grouped together.

There is also inconsistency between the countries listed in the long form and the key countries used in other outputs. (See Table 1)

Table 1: Comparison of Key Countries Used in Outputs

Table 2 shows how these countries compare in terms of visitor arrivals.

Table 2: Number of Visitor Arrivals from Selected Countries

Source: Statistics New Zealand, External Migration Survey

Changes to the Origin of Guest options for international guests will require a redesign and reprint of the questionnaire; respondents will need to be advised of the changes.

The options for origin of guest for New Zealand visitors are Auckland region, Wellington region, Rest of North Island, Canterbury region, Otago region, Rest of South Island, and Don't know.

Although only four of the sixteen regions are included in the options, there seems little need to change these options. Auckland and Wellington regions have the highest populations of the North Island regions and Canterbury and Otago regions have the highest populations of the South Island regions. In addition, the 2000 Domestic Travel Survey shows that 58% of all person nights originate from these four regions.

(3) Electronic Booking Systems

Some accommodation providers have implemented electronic booking systems which may be capable of providing output suitable for the Commercial Accommodation Monitor. (Statistics New Zealand, pers com) However, a review conducted by Statistics New Zealand in 1997 found that electronic collection methods would only be found in the hotel and motel accommodation types, and that there was a low level of uptake at that time. A similar study should be conducted to provide more up-to-date information. This monitoring could be considered as part of Recommendation 95.

Statistics New Zealand has facilities available for respondents to send survey responses electronically. CAM respondents should be given the option of providing data using these facilities - this will need to be evaluated by Statistics New Zealand on a case-by-case basis.

(4) Respondent Burden

Research undertaken by Statistics New Zealand in 1997 showed that respondents who complete the "long form" every month⁷ spend 18 hours per year filling in the questionnaire, and respondents who complete a combination of "short forms" and "long forms" spend 10.5 hours per year. These estimates are expected to be an under-representation of the real time that it takes. (Statistics New Zealand, pers com)

Statistics New Zealand will be introducing a question on "time taken to complete questionnaire" to monitor the respondent burden. The responses should be included in the monthly report to be supplied to the Ministry of Tourism (Recommendation 10). The data should be presented as a distribution rather than provided as mean values.

Statistics New Zealand have also highlighted a possible problem with "respondent fatigue" due to respondents completing the accommodation survey questionnaire every month for a number of years, as well as responding to other SNZ surveys.

(5) Holiday Park Classification

The Holiday Accommodation Parks Association of New Zealand (HAPNZ) has requested that the accommodation classification be updated to separate built accommodation in holiday parks and campgrounds from camping sites, and that statistics for each to be presented separately. Although in principle this could be implemented, the difficulty lies in how to collect the data for these separate accommodation types. Currently each establishment is assigned one accommodation type, and sent one questionnaire a month. To collect separate data for built accommodation and campsites, either the respondent completes two questionnaires (one for each accommodation type) or a redesigned questionnaire is used (which would require significant changes to the processing systems). The former option would increase the burden on the respondent, and the latter would involve considerable costs.

HAPNZ has also requested that the calculation of occupancy rate for holiday parks and campgrounds be reviewed. Currently the occupancy rate calculation includes sites that are closed for the season, as Statistics New Zealand considers that these could be re-opened if the demand arose. HAPNZ feels that calculations on this basis are not truly representative as the campgrounds would not be re-opened. This needs to be discussed further with Statistics New Zealand and the CAM stakeholder group (refer to Recommendation 11).

The Accommodation Type classification used in the CAM needs to be reviewed to ensure that it is a subset of the classification used or the DTS and IVS. Refer to Recommendation 71.

(6) Undercoverage

The current CAM survey population is derived from Statistics New Zealand's Business Frame. The Business Frame includes geographic units that belong to an economically significant enterprise (that is, an enterprise that is GST registered, with a turnover of at least $30,000 per annum), and is updated every year. Geographic units with the ANZSIC codes of H57100 (Accommodation) and part of H57200 (Pubs, Taverns and Bars) are included in the survey population.

Currently there are 3,546 establishments in the survey population (as at March 2002). Statistics New Zealand estimates that there are as many as 7,000 establishments involved in the accommodation industry that are excluded from the CAM. There are only 969 units in this industry that are registered for GST and below the economic significance threshold. The balance are either:

1. not registered for GST,
2. provide accommodation as a secondary activity,
3. provide seasonal accommodation only, or
4. are mainly engaged in renting or leasing their own residential properties or dwellings (other than holiday houses or flats) and are therefore included in Class 7711 - Residential Property Operators.

Substantial undercoverage exists in the CAM survey population for homestays, farmstays and bed-and-breakfast establishments. Extending the population to include farmstays and bed-and-breakfast units would involve adding these businesses to the Business Frame. This would incur significant development costs and ongoing maintenance costs. The work that would be required includes:

1. obtaining lists of establishments to include,
2. cross-referencing with the current Business Frame (to ensure they are not already included),
3. loading the information to the Business Frame,
4. preventing these units from being selected for other surveys, and
5. maintaining the entries (through an annual survey).

A comparison between the Commercial Accommodation Monitor, the International Visitor Survey (IVS) and the Domestic Travel Survey (DTS)), shows that the CAM captures 84% of person-nights spent in commercial accommodation. However, if other accommodation types are also considered, the CAM only captures 32% of total person-nights.

Table 3: Comparison of Person-Nights

Information on the number of accommodation providers not currently surveyed would supplement the Commercial Accommodation Monitor data and provide a better picture of supply and capacity in the accommodation sector. It could also be used to benchmark the sector from time to time (to give an indication of those excluded from the survey population), and could also inform the design of the survey frame.

(7) Current Response Rates

The current response rate target is 75%. For the twelve months to March 2002 the national response rate varied from 77% to 79% for all accommodation types, and from 71% to 84% when broken down by accommodation type. For the same period the regional response rates varied from 64% to 91%. When broken down by accommodation type the regional response rates varied from 0% to 100% - this however was influenced by the number of establishments of each accommodation type in a given RTO area. If the number of establishments is low (for a given RTO area and accommodation type) then one or two establishments can make the difference between meeting or not meeting the response rate target.

The question on employment has a 60-65% response rate.

A low response rate means that there is a higher level of imputation; this will have an effect on the accuracy of the data.

Statistics New Zealand would like to change the target response rate to 80%. This proposed increase would need to be assessed for:

• the impact on timeliness;
• the impact on respondents;
• the work required, and cost, to gain an additional 5% in response rate; and
• the impact on the quality of data.

To determine how the response rate could be increased would require:

• establishing why there is currently a delay in returning completed questionnaires
• identifying the reasons for non-response
• understanding the problems respondents have
• investigating whether anything can be done to overcome any problems identified
• discussing a possible timeliness compromise with users.

This work should be undertaken before resorting to legal action and threat of prosecution. Because respondents are legally required to complete the questionnaires, Statistics New Zealand can take action to prosecute non-respondents. The possible effects of threatened legal action on the response rates, data accuracy and the goodwill of respondents needs to be examined before resorting to legal action. Given the high compliance cost of the CAM, attempts to improve the response rate by taking legal action may have a detrimental effect on the quality of the data, and may also lead to a decrease in the response rate. Legal action should be seen as the last option and used for the most recalcitrant respondents only.

(8) Imputation

Statistics New Zealand imputes missing values based upon data from similar establishments in the same or similar regions. The exception is employment data that is imputed from the Statistics New Zealand Business Frame (which holds employment data as at February each year).

Using employment data as at February each year does not reflect the seasonal pattern of employment in the tourism sector.

From February 2003 the Business Frame will use employee count rather than full-time equivalent employees (FTEs) as the measure for employment. A full-time / part-time split of employees will not be available after this date.

(9) Metadata

Metadata on the CAM is available on the Statistics New Zealand website.⁸

(10) Accessibility

CAM data is available from a number of sources but there is no cross-referencing between sources, and the sources use different names for the dataset. These sources include:

• Commercial Accommodation Monitor report (sent to RTOs)
• Accommodation Pivot Tables (published on the Statistics New Zealand website)
• Accommodation Survey Hot Off The Press (published by Statistics New Zealand)
• INFOS - Accommodation Survey (available from Statistics New Zealand on subscription).

Customised analyses are also available from Statistics New Zealand.

The Commercial Accommodation Monitor page on the Ministry of Tourism website links to the Hot Off The Press, and to contact details for obtaining the CAM report, but does not link to the pivot tables.

(11) Quality and Performance Measures

The Ministry of Tourism is the sponsor of the Commercial Accommodation Monitor but does not currently receive any information on the monthly response rates.

This information will allow the Ministry to identify any response problems that require follow-up action and, in particular, the areas and accommodation types that have low response rates.

The stakeholder group (Recommendation 11) can use this information to design strategies for increasing the response rate.

The monthly report will also include information on time taken to complete the questionnaire, as suggested in Recommendation 5.

(12) Documentation

Metadata aimed at users of CAM statistics is available on the Statistics New Zealand website. However, the Ministry of Tourism also needs confirmation that processes and procedures for imputation, confidentiality and suppression rules, outlier treatment and data capture are fully documented. Such documentation should be made available to the Ministry on request.

(13) Consistency

The International Visitor Survey and the Domestic Travel Survey cover all accommodation types (for both commercial and non-commercial accommodation providers), whereas the CAM covers specific types of commercial accommodation providers only (those that are registered for GST and have an annual turnover of at least $30,000).

(14) Timeliness

Currently the survey data is released about six weeks after the end of the month. It takes about a month to receive 75% of the responses, and approximately two weeks to analyse the data and prepare outputs.

Table 4: Return of CAM Questionnaires

(Source: Statistics New Zealand)

(15) Monthly Origin of Guest Data

The number of establishments, capacity, guest nights, occupancy rates and average stay length is currently collected on a monthly basis. Every third month data on number of employees and origin of guest is collected.⁹

Collecting origin of guest data every third month (rather than monthly or quarterly) is not necessarily that useful because of variation from month to month and year to year. There is a demand for monthly origin of guest data in the CAM.

Options for capturing monthly origin of guest information include:

1. sending a long form to respondents every month. This would significantly increase the burden on most respondents; it is likely the response rate would decrease which would affect quality, and there may be greater use of rough estimates from respondents which would also impact on quality.
2. sending a long form to a sample of accommodation providers. The sampling error, combined with the non-sampling error may mean the data is not sufficiently robust for regional analysis. This option would also incur development costs.
3. add a simplified question on the origin of guest (domestic/international split only) to the short form. This option would need to be discussed with users and would also incur development costs.
4. using some other means of collecting the data, such as cards or a book the guests fill out.

Statistics New Zealand has found that the country of residence information is the most burdensome for respondents to provide. Any changes should not be implemented until the additional burden this could place on respondents is determined, and the reasons for non-response are examined. Any changes to the current system would involve development costs. Respondents should be made aware of the importance of the data they provide, and what it is used for.

(16) Monthly Data Capture

It has been suggested that data capture could be quarterly rather than monthly. However, this would have a number of implications:

• It is unlikely to reduce respondent burden, and may actually increase the burden (as respondents would still be collecting the data but only submitting it at the end of the quarter).
• The timeliness of the data would be reduced, and the CAM would no longer be useful as a timely indicator of regional tourism demand.
• Monthly variations would not be revealed. At the national level total guest nights peaks in January, and is at its lowest in July.

(17) Stakeholder Group

A stakeholder group should be established for the CAM to provide a forum for discussion of issues and developments. This group should include representatives from Statistics New Zealand (the provider) and the Ministry of Tourism (the sponsor). The stakeholder group should meet on a regular basis, preferably quarterly.

6.1.2 Domestic Travel Survey

The Domestic Travel Survey (DTS) provides information on the domestic travel activity by New Zealanders.

Key objectives of the survey are to:

• Provide estimates of the size of the domestic travel market (including both day and overnight trips) to a regional level.
• Provide estimates of the direct expenditure impact of domestic travel in New Zealand, including estimates of the flow of expenditure between regions.

The target population of this survey is all New Zealand residents aged 15 years and over.

The Domestic Travel Survey is a sample survey currently conducted by ConsumerLink, with data analysis provided by Gravitas Limited. A sample of New Zealand residents aged 15 years and over are interviewed using CATI (computer assisted telephone interviewing). To ensure a representative sample, monthly area quotas and weekly age/sex quotas are set. Data is collected on overnight trips made in the last four weeks and day trips made in the last week.

Results from the 1999 and 2000 surveys were published as reports.

Evaluation

(1) Survey Objective

The current survey objectives are:

• To provide estimates of the size of the domestic travel market (including both day and overnight trips) to a regional level.
• To provide estimates of the direct expenditure impact of domestic travel in New Zealand, including estimates of the flow of expenditure between regions.

However, these objectives do not include any quality statements, nor do they reflect the main uses of DTS statistics.

It is proposed that the survey objective be changed to:

To provide reliable, quarterly information on the number, characteristics, region of origin, behaviour and expenditure of domestic visitors. In particular,

1. To measure the amount of expenditure of domestic visitors;
2. To determine the activities domestic visitors participate in, the transport and accommodation types used, and places visited;
3. To provide data for determining tourism expenditure in the Tourism Satellite Account;
4. To provide demographic information about domestic visitors and their reasons for travelling;
5. To provide data about residents who haven't travelled in New Zealand recently and their reasons for not travelling

Target Population:
The usually resident New Zealand population aged 15 years and over.

(2) Questionnaire¹⁰

Three questions (Q20, Q22, and Q78) are not required to meet the survey objectives of the DTS so are not considered necessary to the survey.

Q20 "Did you use a travel agent or tourist service to help you book or plan your trip?"

Q22 "Was there any special promotion or event that prompted you to go at this particular time?"

Q78 "Do you have access to the Internet either at home or at work, or a place of study?"

Information on access to the Internet at home is available from Statistics New Zealand's Household Economic Survey and the 2001 Census of Population and Dwellings. Collection of this information in the DTS may be misleading as access to the Internet does not imply use of the Internet for travel purposes (for research or making bookings). Internet access is also becoming more readily available in libraries, cafes and community facilities, and use of such facilities would not be reflected in the responses to Q78. Alternatively, Q78 could be reworded to assess whether respondents used the Internet for researching or booking travel.

Questions 21 and 46 are not required to meet the survey objectives, but there is demand for information on the use of visitor information centres, in particular profiles of users (and non-users), and satisfaction levels. This demand may be met by incorporating questions 21 and 46 into a supplementary survey; this supplementary survey would need to meet the criteria and use the framework identified in Recommendation 65. Additional questions on satisfaction will need to be designed. How the costs of a supplementary survey will be met will need to be negotiated. This option will need to be discussed further with the Visitor Information Network Board.

Q21 "Did you go to a visitor information centre on this <main overnight/day>trip to <main destination>?"

Q46 "Can you tell me all the places that you got information from on your <main overnight/day>trip to <main destination>?"

Although Q5 and Q14 both ask about date or day of return, both responses are required. Question 5 is only asked for the most recent trip, and the response is used as a check to ensure the date of return is within the recall period. Question 14 asks for day and month of return for each trip, so that the trips can be assigned to the appropriate quarter.

Q5 "What was the date that you returned from your most recent <overnight/day> trip?"

Q14 "Did you return from this trip on a weekend (or public holiday) or on a weekday?"

In the DTS questionnaire there are two "loops" for collecting details about the respondent's travel itinerary. For the trip selected as the main trip the respondent is asked about the places stayed, transport and accommodation used, stops on the way to the places stayed, day trips and activities undertaken. There is a series of questions (a loop) asked for each place stayed. For all other trips the respondent is asked about the places stayed and transport and accommodation used. There is a loop for each trip.

If a respondent has taken a number of overnight or day trips, or the main trip is for a long period then the itinerary looping gets repetitive. This may impact on the quality of responses as respondents will anticipate the questions and may avoid mentioning day trips or stops in an effort to shorten the interview. The DTS data will need to be analysed, and interviewers consulted, to establish how much of a problem the itinerary looping is. Statistics on length of trip and length of interview should also be examined.

In addition, with long trips or trips undertaken a few weeks ago there may be a problem with recall.

One option for shortening the interview length for long trips is to introduce a "cut-off" in the itinerary looping. Respondents with a trip longer than a given number of nights would not be asked about day trips or stops made on the way to places stayed. Table 5 shows the number of overnight trips by the length of trip. A cut-off at 22 nights or 29 nights would affect only a small percentage of the trips recorded. The long trips will need to be examined in more detail to determine the information that would be lost if a cut-off was introduced.

Table 5: Total Number of Overnight Trips (Unweighted)

In the 2000 DTS 16,979 interviews were conducted. Of the 7375 travellers, 5586 had made at least one overnight trip in the previous four weeks and 3087 had made at least one day trip in the previous seven days. Question 60 asks about day trips made in the week prior; responses to this question should be analysed to assess whether more day trips would be recorded if the recall period was two weeks rather than one. More data about day trips may improve the reliability of the day trip data. Consideration would have to be given to the effect on the weighting procedures and weekly quotas, and the increased interview length.

(3) Satisfaction/Motivation

There is demand for information on visitor satisfaction levels of domestic visitors. This is reflected in one of the research themes in the Tourism Research and Development Strategy 2002 (pg 10): "Baseline and tracking research to determine visitor satisfaction levels (to better determine marketing, new infrastructure and product development needs)".

Data users have also indicated a need for information on motivational factors and quality of experience.

(4) Respondent Burden

The Ministry of Tourism does not currently receive information on the length of time taken to complete an interview. It cannot be assumed that the respondent burden is of a similar nature to the International Visitor Survey as the average length of an overnight trip in the 2000 DTS is 3 nights whereas in the 2000 IVS it is 19 nights. Assessment of respondent burden would require analysis of the time taken to complete by both travellers and non-travellers. (See Recommendation 33)

(5) Day Trip Definition

The current definition is a trip made within one day, outside the area in which the person usually lives or works day to day, involving travel of at least 40km one way from home, or travel by airplane or ferry service.

There is no international standard definition for day trips. The World Tourism Organisation (WTO) has not provided a definition for day trips, partly because the definition depends on the nature of travel in the country in question. Instead, guidelines on possible parameters have been provided.

The Australian National Visitor Survey defines a day trip as a trip made within one day, with a round trip distance of at least 50 kilometres and a minimum of 4 hours away from home. It excludes routine travel such as commuting between work/school and home.

The Canadian Travel Survey defines a day trip as a trip made within one day, involving travel of at least 80km one way from home, for all provinces except Ontario. In Ontario the criteria is 40km one way from home.

The definition used in the Domestic Travel Survey needs to be reviewed to determine whether it meets user needs and concurs with people's perceptions of what constitutes a day trip. Parameters that could be considered are a measure of time away from home, and total distance travelled (that is, round trip rather than one way distance).

Note, any changes to the definition of "day trip" will introduce a "break" in the DTS time series data.

(6) Sampling Frame

The target population for the Domestic Travel Survey is the usually resident New Zealand population aged 15 years and over, yet there are no questions in the survey to ensure that the respondent is resident in New Zealand (i.e. has lived in New Zealand for 12 months or more, or is intending to live in New Zealand for 12 months or more).

Consideration should also be given to whether the sampling frame should be restricted to the usually resident population resident in private households. Elimination of non-private households tends to happen by default because of the use of residential phone numbers, but some non-private households may still be contacted. The weighting is based on the number of private households.

(7) Random Digit Dialling (RDD)

When the DTS interviewing was conducted by Forsyte (1999, 2000) the phone numbers were sourced from the Telecom electronic white pages. In 2001 the interviewing was conducted by ConsumerLink using random digit dialling. The RDD scheme used operates as a single-stage number-bank random digit generation scheme, where banks of all number ranges in use are identified and regularly tested and updated. These number ranges are then used to generate random telephone numbers with equal generation probabilities per number. Each number is checked as it is generated to see that it behaves as if it was a connected telephone number in terms of digital telephone network signals, and is not an electronic Yellow Pages published business number.

Numbers are made ineligible if they are a fax number, a business number or the number for a non-private household (e.g. resthome).

Introducing RDD has reduced the level of undercoverage (compared to using Telecom listed residential numbers), as the sample frame now includes unlisted numbers. (Approximately 10-14% of residential phone numbers are unlisted.) However, a far greater number of calls have to be made to gain a contact and this can lead to interviewer fatigue (as there's a longer wait between contacts).

There will always be a compromise between cost and coverage when deciding which system to use. RDD provides coverage of unlisted numbers but at a greater cost; EWP is lower in cost but the coverage excludes unlisted numbers and is not necessarily up-to-date.

With any RDD system there needs to be a mechanism for checking that the ineligible are still ineligible, so that numbers that are no longer ineligible can be re-introduced to the frame.

If EWP is chosen as the preferred option, the provider must ensure that the listing used contains both Telecom and TelstraClear phone numbers.

It should be noted that with either systems there will still be a degree of undercoverage because 8% of households do not have access to a telephone¹¹.

The undercoverage in the DTS is likely to be slightly higher than that shown by the Census results as there may also be households that use only cellphones and do not have a landline phone. The regional sampling regime used in the DTS will not be able to incorporate cellphone-only households as there is no regional allocation of cellphone numbers. In addition, a large proportion of cellphones in New Zealand are prepaid phones, and the phone companies do not hold listings of phone numbers for prepaid phones.

(8) Regional Allocation

The DTS sample allocation is based on the population of each region - the higher the population, the more sample that is allocated to that region. This results in insufficient sample being allocated to Gisborne and West Coast to gather sufficient origin data. The nature of the survey means there is little ability to allocate sample by destination of trip; only an expensive quota sampling scheme might assist here.

(9) Sample Size

In 1999 and 2000 the sample size was 17,000. The sample size was reduced to 12,000 in 2001. With a sample size of 12,000 national quarterly estimates would probably be unreliable for expenditure and person-nights. With a sample size of 15,000-17,000 national estimates would be possible. A sample of 15,000-17,000 would provide 30%-50% more reliable cells at the regional level for overnight trips; 15%-25% more reliable cells for person-nights (and expenditure); and 5%-10% more reliable cells for day trips. Increasing the sample size to 22,000 would add another 5%, 30% and 15% more reliable cells to overnight trips, person-nights and day trips respectively. Refer to Tables 7 to 10.

This analysis is based on data from the 1999 DTS and in particular the report Evaluation of Standard Errors in the Domestic Travel Monitor Survey produced by Alan Lee of Uniservices.

To determine an appropriate sample size one needs to identify some key variables of interest as well as key sub-populations of interest. From the objectives of the DTS, key variables of interest are number of day trips, number of overnight trips, number of person nights, and expenditure per trip. Key sub-populations are defined by the origin of the trip and the destination of the trip. At the least both these should be classified into regions. A further breakdown of interest is producing quarterly estimates - clearly seasonal variation is of interest. Also if ever a move to producing a quarterly QGDP figure for the tourism industry was made then quarterly estimates from the DTS and IVS would be needed. Producing running four-quarterly estimates should be viewed as post survey analysis, specifically a crude method of producing a seasonally adjusted series centred half way between the second and third quarter.

Taking a liberal viewpoint, which assumes that the user of the data is sophisticated and understands and has access to other data sources, one could call an estimate reliable (or perhaps more correctly usable) if the relative sampling error (sampling error divided by estimate) in percentage terms is less than 50%. (Note that in the IVS relative errors in total expenditure for the Tourism New Zealand market shares of 25% are thought to be too high.)

Using that criterion one sees that the current sample size of 12,000 provides few reliable cells for the origin-destination data at the regional level for all of the key variables. Moreover, quarterly estimates of expenditure and overnight trips would probably be unreliable.

Increasing the sample size to 15,000-17,000 would provide, in the origin-destination data at the regional level, an increase of 30%-50% more reliable cells for overnight trips, an increase of 15%-25% more reliable cells for person nights (and expenditure), and an increase of 5%-10% more reliable cells for day trips. National quarterly estimates would be reliable for this sample size. A sample size in the range 15000-17000 is probably the minimum required to meet the major objectives of this survey.

Increasing the sample size to 22,000 produces only marginal gains in the origin-destination data at the regional level: another 5%, 30% and 15% more reliable cells for overnight trips, person-nights and day-trips respectively. It is likely the additional expenditure is not justified for theses key variables, but there may be other variables where this sample size will make a difference. The data is not currently available to determine this.

In the regional flow matrices (region of origin by region of destination) used in Tables 7 to 10 there are 225 cells. With a sample of 12,000 the number of cells with a relative sampling error of less than 50% is 45 for person-nights, 56 for overnight trips and 24 for day trips. Table 6 shows the additional number of reliable cells gained when the sample size is increased.

Table 6: Effect of Changing Sample Size