Evaluating both the effectiveness of interventions and the impact of the law are important for assessing progress, identifying gaps and barriers, and developing solutions and improvement strategies. These may include strengthening the law or regulations, designing more targeted interventions or identifying where additional resources are needed to increase or sustain compliance.
TOP TIP: Evaluation is a cyclical process and findings should create a feedback loop to policy makers, regulators, and agencies to improve interventions, strengthen legislation and increase impact.
- Identify and agree key indicators to evaluate outputs, short-term outcomes, medium-term outcomes, and long-term impact. Some examples of categories include:
- Outputs – Interventions, cost efficiency
- Short-term Outcomes – Compliance, KAP data (regular household or one-off), exposure to SHS/TAPS
- Medium-term Outcomes – Social norm change, prevalence, household economic data
- Long-term Outcomes – Productivity, morbidity, mortality, health spending
- For evaluating inspection performance, include key indicators which measure efficiency, effectiveness, and public trust.
The U.S. CDC has published a guide including 120 key outcome indicators that can be used to evaluate tobacco control programs. The guide includes prevention of youth initiation, limiting exposure to second-hand smoke, and promoting cessation among smokers.
- Prior to implementation, develop a plan to monitor progress and evaluate impact. The plan should be needs-based and feasible. The plan should include the use existing routine data, regular surveys and targeted studies. Find out more about survey types here.
- Identify what resources are needed and map potential sources. These may include incorporating tobacco control questions into existing health surveys or collaborating with universities and civil society organizations.
- An effective monitoring and evaluation plan will help smooth implementation efforts and provide evidence of impact. The data can help set more specific targets, leading to greater efficiency.
- See the case study below for an example of annual smoke-free monitoring and evaluation from Shanghai.
TOP TIP: Situational assessments and gap analyses can be used to inform program planning and enforcement actions.
- Once data has been identified or collected, incorporate findings into action planning. This may include closing loopholes in laws, strengthening regulations or guidelines, and improving implementation activities.
- Utilize findings to identify facilitators and barriers to progress and develop strategies for improvement.
TOP TIP: Evaluating inspections can help identify gaps, priorities, and emerging issues. They can also increase efficiency and effectiveness. Findings can be used to build support of leadership and provide a case for resource allocation.
- Share monitoring data and evaluation findings with key stakeholders to demonstrate the impact of tobacco use and support the incorporation of tobacco control into strategic plans and existing public health efforts.
- Find examples of how data has been utilized here.
TOP TIP: Regularly engaging CSOs and journalists can keep pressure on officials to prioritize enforcement activities, maintaining the profile of implementation and improving accountability.
- Ultimately, the goal of all implementation and enforcement activities is to achieve high compliance with laws. A minimum of 85%-90% compliance across all elements of the law should be the aim.
- Find examples of compliance survey data here.
TOP TIP: Compliance data can be drawn from a range of sources, including inspection data, observational surveys, and studies assessing reported exposure, such as household health surveys. To see compliance surveys conducted by the Global Implementation Program and to learn more about how they are carried out, click the link below.
- Limited resources for evaluation – existing data that are already being collected can provide a lot of intelligence and insights when analyzed and utilized effectively. Partnering with a university or civil society organization can be a cost-effective way of conducting studies.
In 2010, Shanghai began implementation of the city’s Regulations on Smoking Control in Public Places. Tobacco control law enforcement in Shanghai is the responsibility of the Law Enforcement Corps of the Municipal Bureau of Culture and Tourism. The city has incorporated tobacco control law enforcement with safety inspections and encourages voluntary compliance through awareness-raising. Additionally, enforcement officers have begun to use mobile wireless monitoring equipment, such as smoke alarms, to monitor high-violation areas such as fire escapes or toilets. Shanghai also employs a robust public complains system, through pre-established government channels, to engage the public and improve monitoring.
The city works with the Municipal Health Promotion Center and the Shanghai Patriotic Health Office annually to assess implementation of the regulations. The assessment provides compliance data, including the number of observed venues, violations recorded, number of public complaints received, and sanctions issued, as well as key recommendations for next year’s activities. The data shared in the report is a compilation of the regular enforcement reporting done throughout the year by enforcement officials. It outlines successful public awareness efforts and identifies ways to engage the public and city leadership in improving implementation. Click here to read more about the report