Final Evaluation of Future LMS Options

This evaluation was submitted to the CIO and Provost on November 9, 2010. It included the text of this blog and the following summary of our findings.

Summary of Findings

Recommendation: In our review of the possible future Learning Management System (LMS) to replace Blackboard Vista in SPARK, we recommend considering two: Moodle and Blackboard 9.

Moodle would provide the campus with a simpler interface, more local control, and the potential for innovation, but would require a commitment of local resources and a shift in organizational structures.

Blackboard would provide a moderately easier transition for faculty in the short term, but would preserve many of the frustrating aspects of Vista: an awkward interface, a slow response to innovation, and poor customer service.

Comparing Features

Both of these systems offer the basic feature set that most faculty require in a LMS. Blackboard 9 offers a set of features similar to Vista, but this includes the complex and awkward interface that is users’ primary complaint about Vista. Moodle presents an easier interface and similar features, but with a different approach to content delivery and course structures. This would require that some faculty to re-think how their course materials and online activities are structured. Blackboard is a commercial product, so improvements and fixes are made based on corporate priorities. Moodle is an open source product, and can be modified locally if a feature is missing or broken. This made a side-by-side comparison of features challenging because we would often find that, if the basic Moodle installation didn’t have a specific feature built-in, it was often provided by third-party modules, or that there was the potential that we could add it ourselves. Moodle also offers the promise that new and innovative features could be added based on local creativity and local priorities.

Comparing Transitions & Long Term Potential

Both systems will require a period of relearning and conversion. Behind the scenes, both will require redevelopment of the processes that link the LMS to SPIRE and other campus services. The transition to Blackboard 9 will likely require moderately less work in the short term for instructors and support staff, but in the long term this will preserve many of the problems and frustrations we have with the current Blackboard Vista LMS (most notably its awkward interface and the company’s poor track record on customer service.) The transition to Moodle will require more work for instructors and support staff as they adapt to the new way courses are structured, but Moodle has the potential to give the campus more local control over our LMS and the promise that, as a campus, we could do things in Moodle that are more innovative and specific to our campus.

Although Moodle’s interface will be easier for new users, and offers a better workflow in the long run, many veteran Vista users have invested time learning the interface and finding workarounds. For these users, a basic Moodle installation, or a Moodle that simply matches Vista’s features may not be enough to justify the work involved in the transition. We recommend that the University commit resources to provide a Moodle that delivers on the promise of surpassing Blackboard. So the key question the evaluation committee has is this: can the UMass Amherst campus commit to creating the necessary organizational structures, hiring the right people, and sustaining the effort necessary to provide an LMS that elevates our efforts beyond what is possible with Blackboard?

If “yes” then we think that instructors, students, and the University would see a real benefit from a switch to Moodle. If “no,” or if there are doubts, then we think instructors would be frustrated by going through a transition period that didn’t result in a better solution than what they have now, and would prefer that we had selected Blackboard 9, which will provide a familiar, if flawed, experience.

Systems Considered

Four systems were considered: Blackboard 9, Moodle, Sakai, and a “decentralized option” (in which there is no central LMS, just a collection of tools). The decentralized option was discussed early on by the committee and we determined that there were significant functions that it did not provide. Sakai was reviewed fully along with the other options, but the committee was concerned about levels of support and the campus’ ability to modify and maintain it. This left the two options that we are putting forward: Moodle and Blackboard 9.

Moodle

Moodle is an open source LMS that is available for free and can be customized by programmers at an institution. There is a “core” version of Moodle that is supported by a community of institutions and programmers who create modules and submit improvements. There are also commercial providers who will host and maintain Moodle instances for a fee. In Moodle, course content and activities are structured around a single page with a sequence of blocks down the center (typically organized by week, but other options are available).

Interface and Features

The Moodle interface is simple and easy to learn. Its all-in-one-page structure will provide consistency between courses and make it easier for students to find what they need. However, some instructors will see this as a limitation on their ability to customize and create hierarchical “pages” of content. Out of the box, the core Moodle tool set looks more limited than Blackboard’s; however, in our testing over the past few months we keep discovering solutions to what we thought were missing features. One critique of Moodle’s interface would be that some features require forethought and planning in order to do things in the right order.

Many other institutions have already created solutions to the issues we identified in our early testing of Moodle (multiple file uploads, collapsible blocks, gradebook enhancements) so it is hard to know which features are truly missing from Moodle and which exist as modules and which could be created locally. In addition, a new version of Moodle (2.0) is being released in November 2010, which we are just seeing as our evaluation period comes to a close. Moodle 2.0 addresses even more of the concerns that instructors had about features and flexibility. Because Moodle is open-source and customizable, we would have the ability to make changes as we see fit. This has made it especially difficult to compare Moodle feature-by-feature with Blackboard.

The potential for customization is very attractive to certain segments of the campus. Taking responsibility for fixes and maintenance of a basic Moodle would be possible with current resources. However, what is most attractive about Moodle is the potential to go beyond the basic installation and provide instructors with customizations and creative solutions. This would require organizational changes to provide for the intake, prioritization, and implementation of changes. If the campus were able to commit the right level of resources (people, responsibility, priority), Moodle could become an engine for creativity and visionary activities on campus. Contributing modules and modifications to the open source community has the potential for attracting attention to the campus; especially if motivated instructors could attract grants that could be used to develop modules or work on modifications. A closed system such as Blackboard does not offer this potential.

Transition

Like all the options we considered, Moodle has a different interface from Vista, so instructors will have to go through a period where they re-learn how to do what they are used to doing in Vista. The Moodle interface is simpler and more intuitive than Vista, so new users will find it easier to pick up, and users who have complained about Vista’s interface in the past will find it refreshing to use. Users who have invested time in figuring out Vista and are comfortable with way it works may find Moodle harder to adapt to than Blackboard 9, which largely preserves the awkward Vista interface.

Because Blackboard Vista courses are encrypted, there is no official method for transferring courses from Vista to Moodle; however, we are exploring several processes that may make it easier to transfer content and courses.  Of greater concern is that the difference in the way courses are structured in Moodle will require most of the instructors using SPARK to spend time redesigning their course sites to match the new structures. An example of this is the use of “blocks” in Moodle for presenting course content. These blocks are typically organized by week, but other labels are possible. Instructors who are already organizing content by week will adapt quickly, but instructors who have a different approach may need to spend extra time reconsidering how they present materials (the block structure is customizable, but still block-based). The large number of instructors who simply use an LMS to post files of lecture notes and readings will find that Moodle offers many options that are simpler to use than Vista.

Because past transitions have been driven by corporate priorities (when fixes are released, when new products are released, when old products are dropped, and when companies are sold), having local responsibility for Moodle would change the timing. Although some bug fixes and security fixes would need to be applied as needed, the campus could determine its own schedule for adopting new versions and major upgrades to the system.

Behind the Scenes

Moodle is based on PHP-MySQL, which is familiar to our current technical staff. It also runs on servers that are roughly equivalent to our current arrangement for Blackboard Vista. Schools like LSU have shown that Moodle has the capacity to work with large populations of students.

As in the case of Blackboard, we will need to re-write most of the processes that are used to create courses, populate courses with student roster data, transfer grades to SPIRE, and integrate with other services on campus (such as OWL and the Library databases). In the transition period, we would also need to provide additional help to the faculty to assist with the conversion of course content.

In the long term, in order to provide a Moodle system which improves on Vista and lives up to the promise of a locally-controlled customizable system, there will need to be a reallocation of resources and the creation of some new structures to help drive the maintenance and customization.

Moodle would require a comprehensive process for managing the updates and requests for customizations. A collaborative committee of faculty and staff would collect, evaluate and prioritize feature requests and bug fixes for the LMS. A project manager would be essential to coordinating the efforts of the collaborative committee and managing the team that would be needed to design, develop and administer the Moodle system.

Customization and maintenance of the Moodle system would require specific attention to three areas: design, development, and administration. The design tier is not simply responsible for the visual elements, but would focus on coordinating requests from the committee to ensure that they fit into local interface conventions and universal usability and accessibility standards. The development tier would implement, and test new additions and improvements to the system. The administrative tier would be responsible for the maintenance of Moodle, applying patches as needed, and updating existing local customizations as needed.

Blackboard 9

Blackboard 9 is the latest version of the core LMS product offered by Blackboard Inc. When Blackboard purchased WebCT in 2006, WebCT’s Vista LMS became Blackboard Vista. Vista is being discontinued, and Blackboard is asking all their Vista customers to switch to a new “hybrid” product that contains aspects of Vista and the original Blackboard LMS. Blackboard 9 is organized around a flexible template: links to content and activities can be accessed from a list of tools, or can be grouped on “pages” within the course.

Interface and Features

Because Blackboard 9 is based in part on Vista and follows a similar interface philosophy, many users will find it to be familiar to use. However, like Vista, Blackboard 9 has a complex interface that, while it allows flexibility in course structure and design, can make it inefficient to use and difficult to figure out.

A significant question is if Blackboard will continue its track record of not providing timely fixes and improvements. Even the Blackboard 9 interface does not take advantage of many more modern Web 2.0 features that students and instructors are familiar with from other tools such as Google and Facebook.

Transition

Although it was not available right away, there is a tool that converts courses from Vista to Blackboard 9. This should make it relatively simple to move instructors from one system to the other. Some reworking of course materials will be necessary because not all features are supported in the new system.

Because the system is complex, and not identical, instructors will still need to spend time learning the new system and a few will be required to redesign their courses. Many of these changes are in terminology and labels, so it will likely be a matter of learning the new conventions. However, there are other areas where the processes have changed enough that instructors who are used to a certain workaround or technique will need to relearn and rediscover how to do what they need it to do.

Because of Blackboard’s past record on proprietary formats and patent enforcement, it does not appear likely that they will truly embrace “open standards” in the future (despite some claims from the company). Thus it is hard to know for certain how transportable courses will be in the future.

Behind the Scenes

Because Blackboard runs on familiar hardware, our current support group and infrastructure should be adequate for this system. Vista to Blackboard 9 migration tools exist, so depending on the specific courses, transfer of data would be relatively simple, from an administrative, not instructor, standpoint.

Our primary concern in this area is that Blackboard, in the past, has not been very responsive to bug reports and requests for technical assistance. For example: the Blackboard technical staff has been slow to respond to requests for help with integration (in one case, contacting us well after we solved the issue ourselves) and has not been particularly responsive to bug reports (a severe bug we discovered in May 2009 that results in incorrect grades being sent from SPARK to SPIRE in specific cases has yet to be fully addressed) or reports of browser incompatibilities.

Like the other options, we will need to re-write most of the processes that are used to create courses, populate courses with student roster data, transfer grades to SPIRE, and integrate with other services on campus, such as OWL and the Library databases.