In May, we ran a successful Crowdmark EQAO marking pilot at Golf Road Junior Public School. Joseph Romano, who led the pilot at the school, wrote that Crowdmark cut teachers’ marking time in half and provided a smooth, easy-to-use interface to facilitate both individual and group marking from home.
Through the demo, we received additional feedback about the Crowdmark product from the teachers involved in the pilot.
In their words, Crowdmark is the future of marking.
Detailed excerpts follow below. Please note that emphasis has been added.
Responses from the Golf Road Junior Public School self-survey
“The number one thing that stood out to me when using the Crowdmark system was efficiency. Here is a program that I had never seen or used before and literally within minutes I was marking tests as if it was the most natural thing in the world. It was borderline shocking to me how simple it was to use, particularly with no prior knowledge or experience in using anything similar, and how quickly I became accustomed to it. The streamlined interface and straightforward method to marking each section made for quick and effectual way of assessing. Although any modifications needed would be minimal, some things that may improve the program are to better accommodate for multiple choice, true or false, or any other questions that are less subjective as to the grade that should be given (i.e., questions that are either “right” or “wrong” and cannot be assessed on a 4- point scale). As well, it would be helpful to see “real-time” collaboration amongst the teachers who are marking; that is, if one test has been divided into sections, with different teachers marking each section, it would be helpful to see the marks that other teachers have given on the different sections. There is no comparison between Crowdmark and pen and paper assessment. I was able to mark three times the amount of student work in less than a third of the time it would have taken me if those tests had been in front of me with a pen in my hand. The efficiency that the program allows, the streamlined interface, and the simplistic methods of using it all added up to a program that is, simply, the future of marking.“ – Ms. Spirou, Grade 3
Crowdmark is, simply, the future of marking.
“The interface is easy to use. Grading is much faster and easier. I loved how we did not have to handle the exams themselves. <!– Only confusing part was question 6 on paper was not question 6 when marking. A technical glitch that should be resolved easily, I am sure. –> With Crowdmark, teachers are able to mark in their own homes using their own personal devices. There is no need to be sitting in warehouse and marking the exams in person. All exams could be stored on a server. This App has the potential to save time, expenses, and money.” – Mr. Hall, Grade 6
“I liked the feature of easily accessing student work and seamlessly assessing one question at a time. It is easier to complete marking with standards such as EQAO rubrics while looking only at one question from the test for all students. It would be helpful to view when another teacher was viewing/ and or marking a particular student. It is easier, as you do not need to flip back and forth through stapled pages.” – Ms. McPhail, Grade 6
“Favourite Features: – Easy access to question pages (once scanned). It is much easier to mark a class set of a single question, rather than a whole test at once – Lends itself to better/more efficient moderated marking – Tests stored in the cloud (easily retrievable, won’t get lost) – Would be a “powerful tool” for parent/teacher conferences – Allows for partial grades (i.e. 3+ or 2-) – I wasn’t able to see the marks that my colleagues assessed (I’m sure this could be changed in the settings) – Add: sortable, printable spreadsheet of class marks on each question and total – More efficient – Would promote more moderated marking – Useful for parent / teacher conferences – No risk of tests being lost – Easy to call up previous tests / questions – Better for sharing assessment info with next years teacher” – Mr. Lipsett, Grade 3
“I found that the benefit of not having to flip through the actual hard copy document to be the most advantageous element of Crowdmark for me. Being able to just locate the part of the document you need and be able to assess it online made things so much faster and efficient than the traditional pen and paper marking option.
I think that if any changes were made, it would be helpful to be able to see what other teacher’s have marked. If this were to be implemented as an official EQAO marking system, it would save a ton of time if the marker were able to see what the answers had been graded as by the pervious marker. I am sure that it could speed up the process significantly rather than having a room of people mark booklets, only to have them sent to another room to be marked again. Additionally, I think the marking system should change the way a multiple choice question is marked, as it doesn’t make sense to give that type of question. Obviously this is leaps and bounds better than the traditional way of marking. With the movement towards technology, I feel that this is a necessary and useful program for moving forward with contemporary learning and assessing. The benefit of being able to literally input marks in a spreadsheet kind of fashion is amazing. Again, the fact that the booklets or tests of whatever can be scanned cuts down on so many things – even carrying all that stuff home with you to mark for example.” – Ms. Lockyer, grade 3
A separate email exchange
2013-05-28
My name is REDACTED and I teach Grades 5/6 here at Golf Road with Joe. I am writing to give you some feedback about bringing your product, Crowdmark to the EQAO team. I have spent three summer scoring EQAO exams, so I know how frustrating it was to grade exams. Your application, Crowdmark, is significantly faster and easier! You are on to something great here. Joe was telling us today that you have approached the EQAO team with your product. As you are well aware, the Ontario Provincial government has very little money to spend on Education. When putting together your proposal, I would stress the huge cost savings of your product. Here are some numbers to give you a small idea. Each summer, the EQAO team rents a warehouse (Convention Centre) were 1200 teachers and 200 instructional staff are employed. For teachers, the pay is typically $150/ per diem. Teachers typically sign up for a 5 day marking session. Since, your App is so much faster at scoring (grading), teachers might only require 3 days to score. EQAO would save $360, 000 ($150 x 1200 teachers x 2 days) in a given week. EQAO normally stagger the teachers who Grade the various exams over a few weeks, therefore sometimes the Convention Centre is open for a 3-4 week duration. If your product reduces the scoring time by two days each week then EQAO would save $ 1,440,000 over that 4 week period ($360,000 x 4 weeks). Now, think about the convention centre that is rented to house all 1400 EQAO employees during summers. The EQAO team could save a great amount by not paying rent because all exams could be marked at home on teacher’s personal computers. The exams could be uploaded to a central database server and teachers score them at home. Moreover, EQAO would no longer require purchasing the 1200 tablet computers required to grade the exams. In other words, EQAO would not have to replace their Palm Pilots with tablet computers or something similar. As you can see, the potential cost savings for our provincial government would run into the millions of dollars per summer. Any organization would be foolish to not look seriously at a faster, more effective technology, that could save them millions. I’m sure you have thought of most of what I said above, but in case you have not, I give you some food for thought. You have a great product here and we enjoyed trying it out today. I will buy shares when you go public one day.
Crowdmark potentially saves Ontario millions of dollars per summer.
2013-05-29
The information I provided is well known to anyone who has scored the EQAO exams, so by all means share this information with your team members and the EQAO organization. I was telling Joe this yesterday. All I ask is that you remove my name when you share this information with EQAO. Tell EQAO that you received your data through informal conversations with teachers who have written the EQAO exams over the years. Some EQAO grading has already begun (May). Most of the heavy lifting is done throughout the month of July. In other words, the first 3 maybe 4 weeks of July. I do not know exactly how many weeks of testing there are in total. There is no professional organization for the EQAO teachers. Teachers sign up for EQAO directly through the website on a first come, first serve basis. One last bit of information. Some teachers who score (grade) the EQAO exams are not from the GTA. Some are from Ottawa, Sudbury, etc. Those teachers are put up in a hotel for the week and are given daily food allowances. This is all paid for by the government. Second, teachers who commute greater than 40 KM from the Convention Centre are compensated for their travel time. As you can see, your product could not come at a better time for our provincial government. The amount of money saved by introducing Crowdmark, is significant. Your timing is perfect. I would imagine the U.S. market might be even more lucrative. When and if Crowdmark goes public, please let me know. Glad to be of help.