BlogU

  • Toward a Product Evaluation Framework

    By Joshua Kim March 7, 2010 10:43 pm EST

    Part of the responsibilities I enjoy most in academic technology is the opportunity to make recommendations for campus technology purchases. Examples include the opportunity to review and evaluate providers of platforms/products/services for: the LMS, lecture capture, curricular content management, student/faculty collaboration tools, curricular media authoring, synchronous collaboration, mobile learning, simulations, and many more.

    In my experience, the role of a learning technologist in product/company evaluation is seldom as the final decision maker, but rather as part of a team that develops recommendations. It is important that your recommendations are based on a strategic as well as a tactical evaluation. This proposed framework is designed to guide the evaluation process (as well as the writing of requirements), with a goal of moving away from features, design etc. (which will change) towards a more strategic method.

    This evaluation framework is offered not as a final model, but as a route by which I can think through and share some of my ideas around this goal. Conversation, disagreement, and dialogue about this evaluation framework will only improve it.

    The Framework:

    1) Core Competencies: To what degree will running the application, platform or service require your institution to invest in skills outside of your core educational competencies?

    2) Community of Practice: No matter how wonderful the application/platform/service, how great the company, or how attractive the price -- it is almost always a bad idea to be the only (or one of the few) schools adopting the product. Someone has to be first; it shouldn't be you. A critical mass of schools is necessary as we often rely on each other for best practices and fixes, and out of the user community comes the ability to influence the product road map and company support practices. In cases where your institution is determined to be innovative (first), it makes sense to do so within a consortium of peer institutions.

    3) Standards: Is the companies products/services built on industry recognized and open standards? The last thing you want is for the core architecture of the product/service you purchase to become obsolete and unsupportable should your vendor be purchased or merge with another company. A system architected around some proprietary special sauce will be an evolutionary dead-end, as the state-of-the art of development moves along with common standards.

    4) Customization: Any product or service that will require high degrees of customization to work with your current systems (authentication, SIS, etc.) is almost always a bad bet -- no matter how much the company promises to ease your integration path. Today's customization is tomorrow's supportability headache. Far better to live with looser integration and less features than a customized solution.

    5) Lock-In: What happens when your institution wants to move from one vendors platform to another? Will the data transfer? Are files produced by the system standard files that can be stored and transferred to other systems? Will the content that has been produced by students and faculty still be available if the company should go out of business or you should decide to change companies?

    6) Transparency: How transparent is the company in regards to their product road map, operations, and organizational structure?

    7) Ecosystem: Is there an ecosystem of other vendors developing for and around the company that you are evaluating?

    8) Longevity: How comfortable are you that the company will be around for as long you plan to be around? A great free (advertising supported) Web 2.0 platform might seem wonderful today, but may cause big problems down the road if you adopt the platform/service and it disappears down the road.

    9) Business Model: Ask about the business model of the company that you thinking about partnering with. Any company without a rational business model today will not be around to support their product/service tomorrow.

    10): Core Business: Is serving the educational market the core business of your potential partner, or is education a sideline?

    Your evaluation framework may differ -- I'd be interested to hear what you think is missing from the list. What is important, however, is that you and your team develop a methodology for evaluating companies and products that can be consistently applied (and learned from) across evaluation projects.

Go to comments (4) »
Advertisement
On the Job Market Together?

Comments on Toward a Product Evaluation Framework

  • "...it shouldn't be you"
  • Posted by Champion of New Platforms at Multiple Start-Ups on March 8, 2010 at 8:00am EST

  • Joshua, "someone has to be first" and perhaps it shouldn't be Dartmouth, but if every decision-maker in the education community heeded your advice, then no new product would ever gain traction. I'm surprised you didn't say "vendors, let us test your products for free for years and years in order to shore up your community of practice" - or is that tomorrow's column? Even Moodle started with no community of practice and a few users (let's not go down the "free like beer" or "free like puppies" road). So if your team wants to sit back and play it safe (and I understand that LMS and such are critical operations and must be played safe), all the power to you.

  • Someone has to be first
  • Posted by Kyle Johnson , Chief Technology Officer at Guilford College on March 8, 2010 at 9:45am EST

  • As the first commenter mentioned, someone has to be first. What you have to understand is whether your institutional culture and resources are structured to encourage and accept higher risk activities. Understanding that allows you to factor the "first" issue as part of the rubric. You are right that most institutions probably don't want to be first, and almost none what to be first in anything, but being first can allow an institution to differentiate itself. If the product in question meshes well with what the institution feels are its core differentiators, then being first is not only a good idea but preferred.

  • Innovation starts somewhere
  • Posted by Medama , Academic Technology Coordinator on March 9, 2010 at 4:30pm EST

  • I'm surprised, Josh, at how conservative you are coming across in this posting. Ok, not completely, but certainly on some of your points.

    On Community of Practice, I would echo the previous posters. If anything, I think a school like Dartmouth may have stopped innovating 10 or 20 years ago, which shouldn't be the case. As Kyle wrote, "Someone has to be first" and schools with the reputation and research position of the Ivy Leagues, especially, should be able to partner with vendors to help get a product market-ready through use. However, what I might add is a note to vendors...If you want us to be first, you MUST let us in. By that, I mean that if a company wants us to lead in adopting their new product, we either need direct access to the engineers/programmers or to the code. Then, if they choose to keep you from that, you can choose to keep their product away from your campus. Nothing is worse than adopting software without responsive support, without access to the code to fix problems internally, or without an existing community of practice.

    That ties into customization as well. You are right that today's customization can be tomorrow's headache. However, I at least would prefer to adopt a product where customization is possible than where it is not. That's the problem with a lot of commercial products--the end user (institution) is often locked out of adapting it to the local need. Then, when the vendor fails to keep up with a changing IT landscape, the institution is stuck maintaining out-of-date products and runs the risk of losing or keeping a competitive advantage. This, of course, echoes across your framework in terms of Longevity, Lock-In and Transparency, too.

  • 11 -> Empowering
  • Posted by Philippe Robitaille , LMS Programmer Analyst at Sheridan College on March 18, 2010 at 9:00am EDT

  • Consider Empowerment as a likely eleventh element that can lead to closer examination on many fronts. Learners come in all shapes and sizes. So do faculty. Some are creative and some are analytical. Some are detail-oriented and others, not. Some are disabled, others are not. You should have an evaluation criteria with a goal to serving the full range. Will the system be inherently scalable to meet diverse learning needs?

Print
© Copyright 2011 Inside Higher Ed