NSW Population Health Survey: Description of methods 2012–2023

​​​​​​​​​​​​​​​​​​​ ​On this page​

  1. ​Introduction
  2. Survey instrument
  3. ​ Survey sample and sampling frame
  4. Data collection
  5. Data analysis
  6. Response rates
  7. Biases and uncertainty
  8. References​

​ I​​ntroduction

The New South Wales Population Health Survey (NSWPHS)1 is a continuous survey of the health of people of NSW using computer-assisted telephone interviewing (CATI), conducted annually​​ since 2002.

The main aims of the survey are to provide detailed information on the health status of adults and children in NSW and to support the planning, implementation, and evaluation of statewide health services and programs in NSW.

The NSWPHS has evolved significantly in its sampling strategy to adapt to changes in technology and its use with the NSW population over time. From 2002–2011 the survey used a Random Digit Dialling (RDD) landline sampling frame.2 The sampling frame for the survey was altered in 2012 to include mobile phone users, due to the decline in ownership of landline phones and the corresponding increase in mobile phone-only households. For the period 2012 to 2020, the survey used an overlapping dual-frame design that incorporated both landline and mobile phone owners.3

With a steep and differential decline in landline phone ownership in Australia,4,5 the NSWPHS transitioned to a 100% mobile sampling frame in 2021. This page describes the survey's collection methodology between 2012 and 2023, highlighting key changes and their impact on data quality and population coverage.​

​​​[back to top]​

Survey instrument

The survey instruments consist of question modules covering health behaviours, health status, health service utilisation, and attitudes toward health behaviours and service use. The questionnaire is reviewed annually within the Centre for Epidemiology and Evidence and by expert policy groups. Questionnaires for various years can be accessed from the NSW Population Health Survey website. While some questions are collected annually, other questions are collected less frequently, usually on a two or three-yearly cycle. The instrument was translated into five languages: Arabic, Simplified Chinese, Greek, Italian and Vietnamese. 

Survey sample and sampling frame

The target population for the survey includes all residents of NSW. The sampling targets are established at both the local health district level and for children and adults at a statewide level. Respondents for the survey are randomly selected using probability sampling methods to minimise selection bias and to obtain a representative sample of the NSW population. 

Due to an increase in the number of mobile-only phone users in the general population in NSW, a sample of mobile telephone numbers was added to the survey using an overlapping dual-frame design in 2012.3 

The landline phone sample procedures remained the same as in previous years. For the mobile phone sample introduced in 2012, the RDD mobile sampling frame was purchased from a phone number provider including all known Australian mobile prefixes. The provider used proprietary software to test each number to identify valid and invalid numbers. A random sample of valid mobile numbers was then provided to use for the survey. Respondents were selected using RDD of provided mobile phone numbers using CATI and the mobile phone owner was selected. If the respondent had one or more children, one child was also selected at random to ensure that children of people who did not have a landline were also included.

In the overlapping dual-frame design there were three types of phone users: mobile-only, landline only and dual-phone users with a mobile phone and living in a household with a landline phone—who could be selected through either the landline or mobile phone number sampling frames. This required changes to the weighting calculations to account for dual-phone users, who had an increased chance of being selected. To address this, composite weights were used to adjust for the overlap in selection probability which is described in the paper by Barr et al.3

In 2021, the survey moved to a 100% mobile sampling frame, using a segmented dual sampling frame design consisting of list-assisted mobile phone numbers and random digit dial numbers purchased from a phone number provider. List-assisted mobile phone numbers include information relating to the likely postcode of the mobile phone owner and are used to stratify that part of the sampling frame by the local health district. Random digit dial phone numbers do not include any geographical information. No additional​ information was available on the sampling frame.

​​Data collec​​tion​​​

Interviews are completed using computer-assisted telephone interviewing (CATI) throughout the year, from mid-January through to the week before Christmas. Interviews are conducted every day of the year, except Sundays and public holidays, with calls made between 10am and 8pm. All respondents receive a pre-contact SMS, giving individuals who live outside of NSW the opportunity to opt out of further contact.

In addition to directly interviewing individuals aged 16 years and over, the survey also captures information relating to children aged 0-15 years in two ways:
  • as a follow-up interview to one completed by an adult where they consented to further contact (described as a ‘child callback’)
  • as a separate survey specifically targeting households that include at least one child under 16 years of age (described as a ‘child booster’).

​[back to top]​

​Data analysis

The survey sample is weighted to adjust for differences in the probabilities of selection among respondents. Post-stratification weights are used to reduce the effect of differing non-response rates among males and females, local health districts and different age groups on the survey estimates. These weights are adjusted for differences between the age and sex structure of the survey sample. Population data based on Australian Bureau of Statistics estimates and population projections based on data from the NSW Department of Planning, Housing and Infrastructure is used every year to calibrate weights to the population within each local health district.  

The most recent interview data for the NSWPHS (2023) were analysed using SAS 9.4.6 The SURVEYMEANS procedure in SAS was used to analyse the data and calculate point estimates and 95% confidence intervals for the NSW population. The SURVEYMEANS procedure calculates standard errors adjusted for the survey’s complex design. The Taylor series expansion method is used to estimate sampling errors of estimators based on a stratified random sample.7 

Response rates

Response rates measure the proportion of individuals who completed a survey divided by the total number of eligible people who were approached from the sample. The lower the response rate, the higher the likelihood of response bias. Response rates for the NSWPHS are calculated using the American Association for Public Opinion Research (AAPOR) Response Rate ​3 which estimates the proportion of cases of unknown eligibility that are eligible.8 Response rates for the NSWPHS have unfortunately been declining since 2017.  These declining response rates are not unique to telephone-based surveys or the NSWPHS, with similar declines observable across different methodologies (i.e. telephone, mail, SMS) and countries.9  


​Table 1. Response rate (AAPOR RR3*) for NSW Population Health Survey 2017–2023

Year​ ​​Adult ​(16+)
 ​Child booster​ ​
​​

Completed
 (n)

 Response rate
 (%)

Completed
 (n)

 Response rate
(%)
2017​
14,91023.658010.3
2018​
14,6​72
20.870811.6
201914,23617.45357.6
202012,71616.08179.0
202112,91611.74733.3
202212,6779.06342.4
202312,5306.77721.8

* American Association for Public Opinion Research, Response Rate 3.


Figure 1. Response rate (AAPOR RR3*) for NSW Population Health Survey 2017–2023​

​Response rate (AAPOR RR3) for NSW Population Health Survey, text alternative above as a table

* American Association for Public Opinion Research, Response Rate 3.

[back to top]

​

​ Biases and uncertainty

The data for the NSWPHS is collected from a random selection of NSW residents. The probabilistic nature of the selection process reduces the risk of selection bias that would have otherwise resulted from the use of non-probabilistic selection processes, such as those that make use of convenience panels. Sampling introduces some uncertainty to estimates derived from the data, which can be quantified using confidence intervals (CIs), unlike data arising from a convenience panel. A wider CI indicates greater uncertainty in the estimate, while a narrow CI suggests a more precise estimate.

The NSWPHS is also subject to non-response bias. Non-response can occur when a phone number is selected, but the person who owns the phone does not pick up or decides not to respond to the survey. While a low response rate, such as that achieved in the NSWPHS, can be indicative of the potential for non-response bias, the degree to which non-response impacts on the findings of a survey depends on whether there are meaningful differences between responders and non-responders, in a way that is related to the measure of interest.10 Non-response is somewhat mitigated by using weighting procedures to ensure that the sample represents the NSW population on basic demographics.

In addition to non-response bias, the NSWPHS is also subject to the risk of inaccurate responses. Inaccurate responses may arise when a respondent fails to understand a question, fails to recall an event accurately, or feels uncomfortable responding truthfully to a question. Inaccurate responses may arise for questions where the question relates to a behaviour or health issue that is stigmatised or is potentially embarrassing.11 In the NSWPHS, all answers remain completely confidential, and respondents may choose to refuse to answer any question put to them, which encourages respondents to answer truthfully.

These potential biases described above and the potential for other forms of error in surveys should be considered when drawing conclusions and making decisions based on the NSWPHS results available via the​ NSW Population Health website or from HealthStats NSW​​​​​. 


References

  1. ​​NSW Ministry of Health. NSW Population Health Surveys​.  
  2. Barr M, Baker D, Gorringe M, Fritsche L. NSW Population Health Survey: description of methods. Sydney: NSW Department of Health; 2008.
  3. Barr ML, Ferguson RA, Hughes PJ, Steel DG. Developing a weighting strategy to include mobile phone numbers into an ongoing population health survey using an overlapping dual-frame design with limited benchmark information. BMC Medical Research Methodology 2014; 14: 1-9.​​​
  4. The Australian Communications and Media Authority. Communications and media in Australia, Trends and developments in telecommunications 2022-23. Commonwealth of Australia (Australian Communications and Media Authority); 2023.
  5. The Australian Communications and Media Authority. Communications and media in Australia, Trends and developments in telecommunications 2021-22. Commonwealth of Australia (Australian Communications and Media Authority); 2023.
  6. SAS Enterprise Guide version 8.3, SAS Institute Inc., Cary, NC; 2020.
  7. Wolter KM, Wolter KM. Introduction to variance estimation. Vol 53: Springer; 2007.
  8. The American Association for Public Opinion Research. Standard Definitions:  Final Dispositions of Case Codes and Outcome Rates for Surveys, 10th edition. AAPOR; 2023.
  9. Eggleston J. Frequent Survey Requests and Declining Response Rates: Evidence from the 2020 Census and Household Surveys. Journal of Survey Statistics and Methodology; 2024.
  10. Bianchi A, Shlomo N, Schouten B, Da Silva DN, Skinner C. Estimation of response propensities and indicators of representative response using population-level information. Surv Methodol 2019; 45: 231-64.
  11. Tourangeau R, Yan T. Sensitive questions in surveys. Psychological Bulletin 2007; 133(5): 85.
    [back to top]​​​
Current as at: Tuesday 17 December 2024
Contact page owner: Health Survey Program