MAE logo Welcome to MAE!

FALL 2007 NEWSLETTER

The President’s Message

by Meg Blinkiewicz

As I began to draft my first presidential message, I asked several members what they would like to know from a new president. To a person they responded, “I want to know your vision…where you want the MAE to be in two years.” OK, I thought, “the vision thing,” to quote another leader.

My vision for the MAE over the next two years focuses on our membership…being able to give our members—beginner to advanced evaluators, program practitioners, and funders—opportunities to build their evaluation competencies so that MAE can promote evaluation throughout the state to improve programs and policies. I see providing high- quality services to our members as the building block for the MAE to be able to “encourage the use of evaluation findings to improve programs and policies.”

So, where do I want the MAE to be in two years? By 2009, I want our organization to:

How do we accomplish these goals? The old saying, “You have to give to receive” comes to mind. From my vantage point, we are in a place where we, the board, must make a concerted effort to improve member services and to establish strategic relationships with organizations such as Michigan Non-profit Alliance, Council of Michigan Foundations, and the Evaluation Center at Western Michigan University, among others. This is our charge as your elected body. We will report our progress to you throughout the next two years.

I also believe that we cannot succeed without member support. In this newsletter you will learn about the different ways in which you can help the MAE accomplish our goals. Please consider joining a committee so that you can contribute to the MAE and it can continue to contribute to your professional growth.

I’m excited about the next two years and welcome your thoughts and ideas about how we can improve member services and establish strategic partnerships throughout the state.

Calling All MAE Volunteers

As is the case with any nonprofit, the MAE depends upon volunteers to make it work. Please consider joining the MAE team by becoming a member of one of our committees. Your level of participation can be tailored to fit your strengths and availability.

Committee
Charge
Areas of Responsibility
Contact
Membership Raise revenues for the MAE by increasing membership, conference attendance, and conference sponsorships. Sponsorships, grants, and memberships Rita McPhail, Chair
Finance Create stable and reliable administrative structure to facilitate the work of the Finance committee and the MAE. Managing secretariat contract

Brian Dates, Chair
Communications

Provide a quality and useful method of communicating with MAE members and others interested in evaluation.

Increase visibility of the MAE and the learning opportunities it provides.

Communications (newsletters, press releases, complimentary mailings)

Managing MAE website

Brian Dates and Rita McPhail, Co-chairs
Standards Provide professional development opportunities beyond the annual MAE conference to build competencies and promote good quality evaluation. Design sessions for:
  • New or beginner evaluators
  • Funders of evaluation
  • Users of evaluation (practitioners)
  • Intermediate or advanced evaluators

Neva Nahan, Chair
Evaluation Promotion Provide members with annual opportunity to learn, grow professionally, and network.

Establish the MAE as an authority on evaluation; promote quality evaluation.

Develop conference theme

Design workshops

Outreach and advertising

Sponsorships

Meg Blinkiewicz, Chair

2007 MAE Conference Success

What’s Culture Got to Do with It?

by Rita McPhail

Overall (79 percent), participants in the MAE’s 12th Annual Conference, held in March 2007 in downtown Lansing, reported that the conference was quite helpful or very helpful for professional growth. Dr. Rodney Hopson of Duquesne University, an authority on cultural competency in evaluation, served as the keynote speaker. Dr. Hopson addressed how culture competence and diversity add to evaluation theory and practice.

As the second component of the keynote presentation, a panel of Michigan evaluators, funders, and human service providers responded to Dr. Hopson’s keynote address and shared their experiences with cultural competence in the field. One panelist explained the usefulness of engaging a local community member to act as a “guide” when she is working in a culture with which she is not familiar.

The second half of the conference day included further opportunities for learning and discussion for funders, users, and evaluators. Participants chose from the following sessions:

Fantastic, fun, great food and atmosphere are just a few of the comments about the networking session held at 4:00 p.m. after the completion of the last session. The networking session offered conversation, hors d’oeuvres, door prizes, a cash bar, and the opportunity to share and learn about the work of colleagues, either through poster presentations or copies of written work (87 percent of the conference evaluation survey respondents indicated the networking session as an excellent or a very good value to them).

GOOGLE DOCS, ZOHO, and THINKFREE:
A Review of Tools for Online Document Development

by Lisa Marckini-Polk

Your organization’s computers may be networked to a server, and this is great for people within the organization who want to collaborate on a document. However, if you frequently partner with people outside your organization, creating jointly authored documents can be difficult.

Google Docs, Zoho, and ThinkFree are free online tools that allow users to create and edit word processing, spreadsheet, and presentation documents. All are compatible with Microsoft Office, and the documents can be created via your Web browser or uploaded from your desktop. All of the tools allow you to share your document with numerous users, who can then edit the document and save their work, saving you the trouble of coordinating numerous individual revisions. Other reasons you might want to use one of these tools include:

This article provides an overview of the services as well as some commentary on my experiences with them. I tried each with the intent of determining, relatively quickly, the process for using them, what they can and cannot do, and how easy they are to use. I did not try to make the programs show me all of their functionality; rather, the intent was to approach them as a time-pressed potential user and to come to some conclusions about the level of effort required to use them successfully.

Common Process

Each of the tools requires the user to create a free account. Once logged in, you can create a word processing, spreadsheet, or presentation document within the online environment or upload an existing document. If you wish to share your document(s) with others, you will be prompted for the e-mail addresses of those you would like to invite to collaborate with you. These individuals can be provided with read-only access or with read/write access, and sharing/access is managed on a document-by-document basis. Your collaborators will receive an e-mail with an embedded link to the document. Upon clicking the link, they will be prompted to create an account with Google, Zoho, or ThinkFree if they do not already have one. Once they have established an account, they will be provided access to the online document.

Each of the systems saves prior versions of the document and offers some means of reviewing changes and reverting to prior versions of the document. Each also makes some provision for users to add comments, although the systems vary in the usefulness of this option.

ThinkFree and Google Docs allow the user to create folders to store documents in an organized manner. Zoho does not appear to have this option, but word processing, spreadsheet, and presentation files are created and stored through independent interfaces, allowing for “filing” of sorts. Additionally, Zoho allows the user to “tag” or label each document with keywords (as does Google Docs).

Navigation

All three tools are marked “beta.” For me, each tool had some definite bugs, and each required some getting used to in terms of the navigation and range of functions available. For example, ThinkFree’s “power edit” option for Word creates an interface remarkably like the desktop version of Word: if you want to change your font, insert a table, or create a bulleted list, you do it pretty much the same as you would on your local computer. Zoho has many familiar icons at the top of the page, but some of the functions you might associate with Word are not offered. Google Docs clusters word processing tools into folder tabs for “File” work (saving, exporting), “Edit” work (fonts, highlighting), and “Insert” work (tables, hyperlinks); as with Zoho, many functions are not available. In short, if you are an intermediate or advanced user of Word, you may find it takes some time to find your way around these online environments. There are similar challenges with navigation for those features unique to the online environment, such as file sharing and viewing revisions.

Performance

Document design. Each product had difficulties with some elements of my Word document, which I created on my desktop and imported to the online environments. While straight text posed no problem, images, heavily formatted tables, header styles, bullet styles, and embedded MS Excel charts were frequently not exactly like the original document. (Embedded charts were converted to graphics in the best case and empty boxes in the worst case.)

Presentations were universally more difficult to use than word processing documents, and there was no option for creating a slide with an embedded chart in any of the tools tested. Spreadsheets varied in their capacity to create embedded charts: ThinkFree’s chart-building drill walked me through numerous formatting choices, while Zoho and Google Docs were able to create a chart but with fewer formatting and display options. Most of the calculated functions one would want in a spreadsheet were available in all three programs, but it was not always easy to determine how to insert a function, and the programs were often sluggish when I tried to build formulas by clicking in cells. All three programs are able to import and export using common formats for spreadsheets such as MS Excel, comma-separated values, and tab-delimited text.

Overall, ThinkFree is most similar to the desktop versions of Word, Excel, and PowerPoint with which you may be familiar and best able to create documents with the advanced features of these programs. However, in order to access the “power edit” functionality that creates this familiar environment, you may need to download some accessory software.

Sharing with others. Sharing your document is quite easy in all three programs, but none is particularly friendly when it comes to understanding what specific contributions varying authors have made to a joint document. To see the changes other authors have made to your document, you must compare past and present versions of the document. The number of versions grows very rapidly and none of the programs has a very intuitive or visual means of seeing changes over time. Overall, at least in word processing documents, the options for reviewing users’ changes compare very poorly to the “track changes” function available in the desktop version of MS Word.

The programs provide other options for communicating across a team. Google Docs allows the user to insert a comment in Word and Excel documents in much the same way as one would on desktop versions, and this comment is prefaced with the writer’s name. Zoho offers a comment option for word processing documents, but it requires readers to click on the comment to read it, making it pretty easy to ignore. ThinkFree offers a comment option for Excel documents, but for Word documents one can make only global comments related to the document (as opposed to comments anchored to a specific element of the document). Google Docs also opens a chat window when multiple users are editing a spreadsheet or presentation (but not a word processing document) simultaneously.

Conclusions

Each of these tools has something to offer to evaluators working across organizations and in need of a common location for creating and sharing documents. If you are thinking about using them, you should be aware of some ways in which they vary from the desktop versions:

All of the programs offer options for posting documents to the Web for general access, to a blog, and otherwise using technology in new and exciting ways that were beyond the scope of this article. MAE members with an interest in these options would do well to consult www.google.com (select “more” and then “Documents”), www.thinkfree.com, and www.zoho.com.

Dr. Standards

Reporting Results

Dear Dr. Standards:

As a program director in a large multi-faceted human service organization, I have responsibility for an intervention designed to provide supportive housing to the homeless population that the organization serves. One of our concerns has been that some of the clients we serve do not remain in housing placements very long. The intervention employed by the program calls for concerted re-housing efforts regardless of the number of failed placements. Therefore the subgroup of clients not only has several placements, but occupies a significant proportion of program resources, which limits the number of persons that we might be serving.

Almost 12 months ago, we secured the services of an evaluation group to examine the program outcomes in general, and more specifically, identify factors that might account for repeated housing failures among some of our clients. The evaluators used what I considered to be a rather innovative approach. As part of the overall evaluation, they studied data on characteristics of those clients with repeated rehousing histories. This resulted in the identification of several personal characteristics that accounted for most of the reason for repeated housing failures. Measures of these characteristics were used to develop an equation that can be used to identify similar clients at the outset of services, which will allow us to take extra program measures “up front” to make placements more permanent. The equation helps us to determine the relative likelihood of our individual client estimates, so we can concentrate on those clients who almost certainly will have difficulty.

I asked the supervisors of the housing programs to review the preliminary report and to make comments before it was made final. I have some background in evaluation; however, my supervisors have limited, if any, experience in the matter. Generally their comments indicated their overall lack of understanding of the information and of how to interpret/use it. I was hoping for a chance to have input into a rewritten draft of the report, but it was submitted to the Board of Directors almost as soon as the supervisor comments were received.

The report is very long and contains numerous charts, graphs, and data tables. The supervisor comments were included, un-edited, as an appendix. The last section of the report is a four-page summary of findings. Because the document is so long and technical in nature, the Board of Directors has chosen to employ the summary for decision-making. As a result my supervisors and I have to prepare an entire “plan of correction” for difficulties that the Board deems quality problems. In the past, I’ve had good experiences with program evaluation and was hoping to convince my staff that evaluation could be useful and positive in nature. We all feel more of a “got you” at this point. Is there anything that I can do?

The evaluation you describe, while positive in many arenas, clearly lost sight of Utility Standard U6: Report Timeliness and Dissemination of the Program Standards. This standard mandates that significant interim findings and evaluation reports should be shared with the intended users of the evaluation in a manner which allows them to make use of the evaluation in a timely manner. The evaluation group did offer the supervisors the option of providing feedback as Guideline H suggests (checking draft reports as appropriate with representatives of the intended users for clarity and factual accuracy in order to prevent premature release of findings). However, they were unsuccessful in employing the guideline as a whole by failing to take the feedback into consideration in the final report and releasing it prematurely as a consequence.

The poor timing of the report release prevented any thoughtful review of the findings, which in all likelihood would have revealed the failure of the evaluators to incorporate an analysis of the supervisors’ comments into the discussion. This might have signaled the presence of other difficulties, e.g., the implications that quality was an issue. They also failed to take heed of Guideline I, which advocates considering a variety of methods for dissemination of findings (summary reports, audiovisual presentations, panel discussions, etc.). Difficulty with Utility Standard U5: Report Clarity is also an issue in this case. Guideline F informs the evaluator to use technical language sparingly and help the stakeholders understand technical or unfamiliar terms by providing a glossary, separate summary and technical reports, and training opportunities for stakeholders. Finally, the evaluators lost valuable information by not analyzing the comments which the supervisors submitted. This violates the intent of Accuracy Standard A9: Analysis of Qualitative Information, which promotes appropriate and systematic analysis of qualitative information so that evaluation questions might be answered effectively. The essence of this standard is to enrich the understanding of program phenomena so that faulty conclusions are avoided. In failing to analyze the supervisors’ comments, a valuable part of the evaluation was omitted.

With respect to your question regarding what action might be indicated under these circumstances, I would suggest using this response as a guide to the applicable standards. It may benefit you to look with more depth at these, and perhaps other standards, and to prepare a brief report which you could share with the CEO of your organization, or perhaps even the Board of Directors.

If you wish to submit a question to Dr. Standards, send your evaluation dilemmas and questions to bdates@swsol.org. Dr. Standards will respond via the newsletter column. You may order The Program Evaluation Standards, 2nd Edition (1994) from Sage Publications, Thousand Oaks, California, e-mail: order@sagepub.com.

Current MAE Board Members

MAE President Meg Blinkiewicz Welcomes New Board Members

Just in case you were not able to attend the MAE’s annual meeting held during our annual conference, I want to introduce you to our newest board members listed below.

In addition to our new board members, the MAE has a new Secretariat contract with Jacqueline LaFay, Public Sector Consultants, 600 W Saint Joseph St., Suite 10, Lansing, MI 48933; 517/484-4954; jlafay@pscinc.com.

Mike Acosta

Mike Acosta holds a BS in Psychology from Michigan State University and an MSW from Western Michigan University. He has held positions with the Wayne County Department of Social Services, the Michigan Department of Social Services, and the Wexford Missaukee Intermediate School District, and spent time as the director of a private nonprofit substance abuse prevention and treatment program in Cadillac. Since retiring in 2004, Mike has worked as a consultant; he also specializes in grant writing, which has led to his current interest in evaluation. Mike is keenly aware of the need to implement evaluation to demonstrate program effectiveness. This led him to participate in the Michigan Association for Evaluation’s “Engaging Minority Evaluators Project” (EMEP). Mike hopes that by serving on the MAE Board he can give something back to the organization while remaining both a student and promoter of evaluation.

Elizabeth Agius

Elizabeth Agius has been a program evaluator and researcher at Wayne State University for 12 years in various departments. For several years she has been providing programming and evaluation direction to New Detroit, Inc. for its community-based organizations. This work has included an evolving model of capacity building and the development of an evaluation framework. Currently Elizabeth is focused on building the capacity of small to mid-sized organizations through a grant from the federally funded Compassion Capital Initiative. Her experience includes participatory, multi-methods and longitudinal designs that are developed in close concert with project stakeholders. Evaluation findings are communicated to various audiences and have been disseminated at local, regional, and national conferences for academic and community audiences. She has co-authored several publications on evaluation of school-based and community-based programs.

Sharon L. Anderson

Dr. Sharon L. Anderson has more than 25 years combined experience in higher education administration, as a consultant in program design and evaluation, and in philanthropy. She has administered programs at Michigan State University, Tuskegee University, Western Michigan University, and the Kellogg Foundation. Currently, Sharon holds a position as Community Investment Officer at the Kalamazoo Community Foundation. She founded Praxis International LLC in 1989, a consulting firm specializing in program design and evaluation and has consulted on projects locally and internationally. She is a member of the American Evaluation Association, the American Association of Grant Professionals (AAGP), and the association for Research on Nonprofit Organizations and Voluntary Action (ARNOVA). Sharon was a participant in the Emerging Minority Evaluators Project (MAE), 2004–2006.

Rita McPhail

Rita McPhail joined the MAE Board in 2005 and chairs the Fund Development Committee, which includes producing the newsletter and revamping the website. She is working with the MAE Secretariat to make the newsletter look more professional and to convert it to an electronic version. In addition to the newsletter, she is committed to finding links and lists of appropriate organizations and individuals to whom the MAE can forward the newsletter and website link. She holds a BSW and an MSW and is recognized as a LMSW by the State of Michigan. She started out as a clinician in the mental health field and gradually became involved in program development, evaluation, grant writing, management, and administration with other nonprofits. Rita is the sole proprietor of RMK Management Support, which was established in 1993. Her mission as a consultant is to help organizations and people grow.

Kurt Metzger

Kurt Metzger is the director of research at United Way for southeastern Michigan. In this role he is responsible for developing the research structure that will allow the United Way to move ahead with its Community Impact Strategy. This effort is designed to combine primary and secondary research to identify the key social and human service issues facing the residents of southeast Michigan. Prior to joining the United Way, Kurt directed data services for the Detroit Regional Office of the U.S. Census Bureau and was director of the Michigan Metropolitan Information Center (MIMIC) at the Center for Urban Studies at Wayne State University. Kurt serves in a number of state and local organizations, including the Michigan Land Use Leadership Council, the Governor’s Lead Task Force, Michigan’s Children, Michigan Kids Count, the Michigan Cool Cities Initiative, the Michigan Nonprofit Association, New Detroit, Detroit Renaissance, and the Detroit Regional Chamber of Commerce.

Jane Powers

Jane Powers is a senior consultant for health policy at Public Sector Consultants (PSC). She conducts research, evaluation, and analysis on a variety of health policy issues. She provides consultation and facilitation to partnerships, task forces, coalitions, and health care organizations; conducts focus groups; helps develop and analyze health care surveys; and monitors state and national public policy activities. Prior to joining PSC, Jane held several policy planning and analyst positions in the Michigan Department of Community Health (formerly Public Health). In her work in the department, she provided consultation for the development of primary care centers throughout Michigan, coordinated development of Healthy Michigan 2000, managed creation of the department’s Center for Healthy Infants and Pregnancies surveillance system, developed guidance for the statewide community health improvement process, and provided analytical support for the Comprehensive Health Plan Division. Jane holds a bachelor of science degree from Michigan State University in public affairs management with a health emphasis.

Deborah A. Willis

Deborah Willis has more than 30 years in the human service field both as a practitioner and as an evaluator. Early in her career she delivered behavioral prevention services to children and families in New York, and taught in Queens and the South Bronx. She has a MSEd from Northern Illinois University, an MSW from the University of Michigan, and a Ph.D. in social work and sociology from the University of Michigan. Deborah joined the Guidance Center as the director of the Center for Excellence in September 2003 where she served as the liaison between university researchers and the Guidance Center and directed evaluation and training for the agency. She has taught research and evaluation to MSW students at Eastern Michigan University and the University of Michigan for the past 10 years. Deborah currently serves as the director of programs at the Guidance Center and continues to oversee the agency’s evaluation projects.

SAVE THE DATE: MAE 13th Annual Conference

March 31, 2008
Lansing Radisson Hotel

Program: “Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs”

Plan to attend the 13th Annual MAE conference featuring Dr. Melvin Mark of Penn State University, editor emeritus of the American Journal of Evaluation and past president of the American Evaluation Association, as the keynote speaker.

After his initial presentation, Dr. Mark will entertain questions from the audience.

The second half of the conference day will include further opportunities for learning and discussion as we will offer several workshops, including:

Additional workshops are in the process of being developed. Stay tuned for further updates!

Professional Development Opportunities

Systems Concepts in Evaluation Workshop

Presenter: Bob Williams
Date: Tuesday, December 11, 2007
Time: 1:00 p.m–5:00 p.m.
Location: Kellogg Hotel & Conference Center, 55 S Harrison Rd, East Lansing, MI 48824
Cost: $50 MAE members, $65 non-MAE members (Coming Soon: On-line registration and payment)

Concepts and methods drawn from the systems field are increasingly seen by evaluators as valuable additions to their toolkits. However, after the first step or two, many evaluators find themselves faced with a bewildering array of principles, methodologies, concepts, methods, and techniques. All sorts of questions arise, such as:

Bob’s workshop addresses these questions using a case study and by introducing a couple of systems methods that provide useful additions to the evaluation toolkit.

For more information and updates please visit www.maeeval.org/ProfDev.htm.

This workshop is co-sponsored by the Michigan State University Outreach and Engagement and the Michigan Association for Evaluation.

 

MAE Member News

New Newsletter Feature!

As a service to our members, the MAE is adding a Member News section to our newsletter. Although viewed by our larger audience, only current MAE members may post information. If you want to share the news of a recent promotion, project, achievement, training, contract, or any other professional milestone, please send the below listed information to the MAE Secretariat at info@maeeval.org in order to be included in upcoming newsletters.

Call for Articles

The MAE newsletter includes articles on evaluation theory, practice, and methods, as well as articles on related technology and tools, to assist evaluators, funders, and program stakeholders in their work. Articles should be original works or previously published with documentation of permission to reprint, 500 to 1,500 words in length, and submitted electronically. Articles may be submitted at any time for consideration by MAE; the person submitting the article will be notified prior to its publication.

Please submit your articles to the MAE Secretariat at info@maeeval.org.

Newsletter Feedback

Please let us know how you like (or dislike) the new format of the MAE Newsletter. Please e-mail your comments to info@maeeval.org.