All right folks we are going to go over procedural evaluations today. So in terms of the outline, we've got an introduction, we'll talk about evaluability. The ability to evaluate a program or different aspects of it. And describing the program and logic models. And these constitute procedural evaluations. Now earlier lectures noted that the need to describe the program was important and this lecture details exactly how this should be done in the context of procedural evaluation. So procedural evaluations are basically assessments of the processes and employed by a program, they are often conducted when the aim is to improve efficiency, when outcome data is difficult to obtain, or when the evaluation is focused on an emerging program, and when funding is not available to conduct an outcomes assessment. So these are typically for earlier stage programs or for smaller scale endeavors where improvement is sought. However the level of resources available is less than would be necessary for rigorous outcomes based evaluation. So, the first step of procedural evaluation as describe the program. The C.D.C. provides the following checklist describing programs. Compile a comprehensive program description including needs targets outcomes activities and resources identify the stage of development and context of the program to convert inputs activities outputs and outcomes into a simple global logic model and to elaborate the model as needed and to develop more detailed models from the global model as needed. The first consideration is evaluability so Rossi et al note that it is important when commencing an evaluation to also assess the ability to which it can be evaluated so three activities are involved. The description of the program model. And assessment of how well the program logic is defined. And identification of stakeholder interests so again we're coming back to this idea of the role and influence of stakeholders the point that the Rossi crew is getting at here is that you have to look and see what kind of thought process lies behind the program some programs are operating the way they are simply because that's the way it's been done last thirty years some of them do not have formal mission statements or formal designs or they have never actually considered a logic model so there are some things in which the ability to evaluate the program with procedural aspects is quite limited now. When you're thinking about this you need to think about three objectives so the goals objectives and needs are well defined. Goals and objectives are plausible. And the relative performance data can be obtained so this is what you're talking about with the evaluability so the program is operating but they really don't know what their goals and objectives are they're not well defined, it's vague it's rather difficult for you to come in until you have sound valuation on the process because you need to link close to the goals and objectives so these are defined it's rather impossible to evaluate that program optimum. Now is these goals and objectives are not plausible for instance if someone has we want to alleviate all pain and suffering in a population. But it's virtually impossible. That's hard to evaluate so you need to make an assessment of that offset now also relevant performance data can be obtained, you can have something that tells you whether or not these processes are going on in an optimal manner. So you have to look at whether you have information about how the program operates with basic data process so you need to look into these things as you begin to commence your evaluation. I also have intended users of evaluation results I have stated how they will apply findings. Okay so this is another objective in terms of now you believe. And what they're trying to get out here is that somebody has to apply something or be willing to use or implement the conclusions of the evaluation. Now, if the prior objectives have not been met your role as an evaluator is to communicate this to the program staff and or the evaluation sponsor and work with them instruct well designed and plausible goals and objectives, identify relevant performance data that can be obtained and to communicate with intended users of evaluation results to understand how they will apply findings so the point is that if you're working with a program that really has not formalized their their mission they're goals their objectives and they're operating in a rather a an absence of these types of formal stated facts of what they're targeting. You need to help them and you're the product of your evaluation at this point would be to formalize strong mission goals, objectives, statements and to help them see what is plausible within the amount of resources that they could be collecting in terms of process or performance data so that they begin to have a setting in which improvement is possible by way of evaluation so that's why you really need to look at the stage of the program and really get a good idea where it's at before you really design your methods because if you don't take these things into consideration the methods that you would propose could be far out of line with which the program is developed at that point. You know helpful questions include is the mission and goals clear is the target population clear is their agreement about intended effects so these are things you should be asking about the program and also is it plausible that program activities would achieve the intended effects so again if you want to alleviate all pain and suffering in the world I'm not going to do that with one small program such as you know less than one percent of it. Describing the program. So once you've ensured that evaluability objectives have been met, you must map out the program logic. Now some will call this the program's impact theory, it's a fancy way of just saying what is the program supposed to be achieving. Now more common in public health is the use of the term logic model, particularly with the C.D.C.. Now program impact theory are the assumptions about how the program leads to improved outcomes so this includes understanding the service utilization plan: how the public will use the program; the organizational plan: how the program will be organized to interact with the public. So together these two plans describe the process of the program. Now. The diagram for program theory in terms of the Rossi text is to have a service utilization plan. You basically have the target population. Interaction with the delivery system and then when the program organization plan here on the bottom you have the program facilities personal activities and then you have the program target service transactions so you bring these things through and bring them to these outcomes and so you're trying to achieve that it's not as tight as the logic model as you'll see that's typical in public health as we proceed. Logic models which are much more commonly used are diagrams as follows you'll have inputs, activities, outputs, and then you focus on what happens so you see the short term effects intermediate effects and long term effects or outcomes so inputs would be resources that go into the program and on which it is dependent the mount its activities, activities would be actual events or actions done by the program and its staff. So this is a very simple logic model but this is just the basic conceptual underpinning of what you'll need to do when you create your own logic model. So, you'll have to know the resources that are available you'll actually have to know what the program is doing. Now if you have selected a very large program for instance all of Medicaid. This could become quite a task to try to put together a detailed logic model which is the reason why some smaller more feasable activities may be good but you know it's not impossible to do larger programs. Now the outputs are the direct products the program activities often measured in countable terms so the number of sessions held for instance and now the outcomes are the changes that result from the program's activities and outputs often in a sequence express the short term intermediate and long term. Now the C.D.C. provides an example using the childhood lead screening program. So the inputs they list involve funds, trained staff for screening and clean-up, relationships with organizations, and legal authority, now their early activities include outreach screening and identification of early blood levels in children. Now the later activities involved case management referral to medical treatment environmental assessment and referral and family training so this is really the process with which the program operates but then the outputs in terms of the pool of eligible children, their screened referrals, the number of leaded homes that are identified and the number cleaned-up. So the early outcomes are lead source identified, gets eliminated, families adopt in-home techniques, and and if with a need in pain his adopted home techniques and the blood levels get with elevated but with blood levels get medical treatment. And then the later outcomes is that elevated blood lead levels are reduced and developmental slide is avoided and the quality of life has improved. Now once aspects are listed their linked by arrows in a causal map. So you have the activities and the outcomes so again here you have the outreach screening identification of the children and then case management. Now once you go to case management you have this environmental assessment train the families and refer for medical treatment so these things lead to being alleviation of the source of lead the family learning techniques that will prevent lead from recurring and then you also have the medical management that will actually reduce the blood level concentrations. Which is happening here and then the development and intelligent will improve and then more productive and or higher qualities of life here. Now you also have other examples and this is the simple logic model surveillance research and development, these things lead to capacity building, communication, partnership, and leadership together these lead to change in physical environments and change in social environments and this is prevention and or control of the problem so again the surveillance is the screening the identification. Now this research and development is a bit of a different terminology here but we're basically looking at capacity building and you could look at the way the families were trained and their capacity to avoid further exposure to lead, communication, partnership, and leadership. You know these are different terminologies that would be used to talk about what is going on in the interaction with medical providers and so fourth and the change in physical environments is the change in removal of lead in contaminated homes and the change in social environments would be going to change in the different behavioral functions that are associated with the lead elevated blood levels and the prevention and control of the problem would be when these things are alleviated and the behaviors and actions moving forward are such that exposure will be unlikely subsequently. Now complex examples of logic models is something like this now again you have the overarching activities outputs and effects but now you're really taking the surveillance days about the same but now you're really breaking up research and development to what exactly that means and how you're developing interventions testing interventions how this is happening. Developing evidence based strategies policy change is adopting them whether or not that happens all is proceeding changes in physical environment but you also really state what capacity building is how this works communication partnership and leadership and you notice that you can have multiple lines for instance going to the activated constituencies for prevention and shared vision. That things start to get a little bit more complicated now this of course takes more time to do than a simple logic model however, the more time you put into your logic model the clearer your thoughts about how that program operates and the easier it will be to propose a sound methodology for evaluating that program particularly if you're proposing the process evaluation procedural evaluation. So while this may be cumbersome to really break down think exactly how a program operates you could pick out a number of things that you could measure a number of different activities that you could put aims about how you're going to assess and then it would be easier to think of what kind of procedural data would be necessary for instance the number of policies changed the number of tools developed. Attitudes and behaviors assessment. How many key groups have participated or adopted different strategies. You can go on and on and by looking at these different boxes now is going to be more difficult to identify and these opportunities to evaluate if you have a very simple logic model so it's worth it to put some thought into it upfront because if you can figure out how it's operating and why and what the attended effects are it's going to be very difficult for you to uphold a sound evaluation strategy so you may as well put some effort into your logic models. And the C.D.C. offers several questions to guide the development of logic models. Basically look at utility feasibility propriety and accuracy so in terms of utility thinking about the model how it will be used the level of data appropriate or is there too much or too little detail is the program description intelligible to those who need to use it to make evaluation planning decisions this last one is important is it intelligible to those who need to use it to make evaluation planning decisions now the feasibility is does the program description include at least some activities and outcomes that are in control of the program very important. If your logic model is highly conceptual but nothing is actually in the direct control or domain of the program it's not going to help you figure out how to evaluate that program because if you're talking about things that it doesn't actually control at least in part it's going to be difficult to use that watch model effectively to guide you to a decision now propriety is they evaluation complete and fair in assessing all aspects of the program including its strengths and weaknesses and does the program description include enough detail to examine both strengths and weaknesses and unintended as well as intended outcomes one here is to be thorough and to not have an I wouldn't say a biased but you know you're dealing with people that can be impassioned in a number of different opportunities for evaluation and there can be a tendency to avoid emphasizing weakness you see it from time to time so the point here is that you're not doing anyone a favor if you're shying away from clearly addressing and identifying measuring recording and you know focusing on weakness as well as strengths it becomes a balance so the point here is to think broadly and think in detail about both the strengths and the weaknesses of the program. Now accuracy is is the program description comprehensive, have you documented the context of the program so that likely influences on the program can be identified now again seems to be alluding to stakeholders but really any influence on the program if you're not aware and considering the influences on the program your ability to understand how that process is working and operating is less than is necessary to really come up with something that is actionable for you to come up with something that can actually improve the public health. So procedural evaluation basically consists of evaluating goals and objectives functions components and activities login linking activities to goals. However it is not enough to look at the logic model of a program you must investigate whether the program actually functions and a manner that corroborates the logic model there's a number of programs have a fantastic logic model that gets them some funding to move forward but then once you really look underneath that fail to see what is actually happening on the ground it is unoperating in that manner so one thing to assess these goals and objectives and the conceptual underpinnings of the program you have to go further and look at how are things actually happen. Now evaluations that identify programs that are not functioning as intended primarily strive to change the program operation bringing it in line with the assumed logic and align with the assumed logic.. Of programs that are operating as intended undergo further assessment intended to answer is the program logic an appropriate response to the needs of the target population need identifying during the session so again that they're not operating as intended and they undertake that it first or you start talking about is this an appropriate design an appropriate formulation work program given me it's very easy to say whether it was carry forth in a manner as intended at the outset of the program. How many procedural evaluations focus on service utilization coverage So basically you're looking at are the intended targets reached and bias or specific groups of targets missed So you're looking at the OK you're supposed to provide this service to these people do you reach them are they benefiting from your program are the characteristics of the population that is receiving services that which you have targeted at the outset of the program. So good procedural valuations should a SAS program goals and objectives map and assess the program logic. But investigate whether the program operates as intended an answer of whether the program process could be improved and or mean more if it. So successful procedural evaluations should result in clarification of goals objectives and logic in a restructuring of program activities to run more efficiently and or better meet target population needs you can't so it's not the Seadrill evaluation is just basically you have a goal and mission statement are these things out there it's actually looking and fine detail how the program is structured and operating and looking for ways to gain efficiency and it doesn't mean just to slash budgets or propose that you don't need certain programs that so that it's basically to look at how the program is operating with all of the resources and then figuring out given all of those resources can there be any room for improvement how the process is carried forward therefore you have additional kind resources funding and or to gain more incremental improvement and population as we are able to and I strongly believe there is always always always opportunity for every time you look there should be something that you can find or propose that's on as long as you matter how much effort you put into thinking and. How strongly you. You set forth to identify these things and think Rotman and. Now common proceed through the valuation questions including how many people are touched by the program I mean people actually receive services in some way or another or benefit either directly or indirectly. Are those touched the intended targets so to reach you're trying to get to. Our targets receiving the proper amount tight and quality of service it's not that means that the needs are remaining and to be. Are there under represented population sidedness among the service population you're supposed to reach an entire population but you don't reach those who are elderly for instance because they don't come out they have mobility issues that inhibit them reaching the access point of your program that's something you need to think about and that's something that you could offer an opportunity or a strategy for so you can think of various populations like Nancy who. May not be reached. Is a target population aware of the program that's a big one is often we have awareness issues where those that are in the greatest need in our Those least amount. Cation as to the opportunities there hordes and I'm. Programmed. Now is staffing an organization sufficient to carry out the program I was free source issue was a difficult one as of course additional funding always helps he's a thing to say well you need to get additional on what I'm talking about is thinking deeply if that's the case if you find that a program's Basseley understaffed or just doesn't have what it takes to carry out. Then you need to proposal and action step that is very specific for instance you could say well you need to go out here funding in terms of constructing a database that will so it's a follow up and identification outcomes and in order to redesign or restructure or realize better opportunities for. Specific things about how they go about things not just that they need more funding how to go about it opportunities to do so. Now are resources used efficiently Now this comes in a lot where you look at your home use of automation and use of your programs you so staff and how this works is this. Mission I'd say more often than not you can always find a way to more efficiently use existing programs. How is the program in compliance with the clickable standards space but it looks at how other programs are structured and how funding. Federal organizations for instance how they require programs to operate for everything in terms of data safety. Human Subjects protection from Asian protection you know these are and lot of things they still need to think considered to avoid risk they are not raking in mind these things. And also our participants satisfied with their interactions with the program now you'll see this a lot in health care. Reform is a valuation they use a lot to say about the satisfaction levels hospitals for instance so there's something to be said about that. So your sign now is in four pages or last answer the following question and post your response to the discussion or situations in which procedural evaluation will be proper compared to outcomes evaluation If so then describe some examples if not then depend your answer so I'd like you to. Post this has a discussion about it now that I think looking over your respect this and getting back to each of you. So now we're going to start moving insists more specifics about methods for this are going to need to be thinking about what methods you will pose in your final project so next semester project is your literature review you're going to do a formal literature review you know how to do that and you're and I go through that process and then you're going to propose your evaluation by end of your final We're going to go we're going to steps of considerations from here throughout the semester. From this we need to take home with you in your head to think about logic models how well I write a lot of my dough or. Or my program that I selected and I very well may make that its own separate sign and. One of the next modules now after you've gone through this week's lecture. Cheerios but I may have you sir submit your logic models ahead of time so that you can make sure that everyone's mind understands how to do this because it is how cool once you go to those you're not that's you have any questions please let me know. Thanks and best wishes.