Crowdmark provides real-time insight into grading activity. Real-time means that at all times, the data displayed in the assessment dashboard is current. We track a velocity (evaluations per hour) metric related to questions as a whole and also for each grader within a particular assessment.
How it Works
In order to support real time analytics, Crowdmark stores computed grading activity over intervals of time in a data object we call an evaluation timing cluster. We chose that name because we are analyzing what happens while the grading team performs “evaluations” of student work. We coalesce and store the data measuring grading activity over a period of time into “clusters”.
When a grading action happens such as drawing a freeform annotation, adding a comment, or assigning a numerical grade there is a side effect which we call a touch. A touch event takes in the thing in the assessment that we’re timing (a question, or a grader’s membership in an exam) and looks in the database to see if there is already a timing cluster started for that thing.
As time passes, new events can be added to a cluster. If a cluster is found for the current timeable thing, that cluster’s ended_at attribute is updated to the current time. This means that as grading actions occur, the cluster tracks a longer and longer time interval. The duration of the cluster is the difference between ended_at and started_at attributes of the cluster object.
If there was no cluster found for the timeable, a new one is created with a started_at attribute set to two minutes in the past and it’s ended_at set to the current time. This is because we make the assumption that the grader spends approximately two minutes on the question grading it before actually taking a grading action. The arbitrary selection of the two minute threshold appears to be effective but is not yet optimized.
Clusters can stop growing in time. A timing cluster can expire. That is to say, if a grader stops grading for a period longer than MAX_GAP (ten minutes), the cluster is no longer extended and a new one would be created. This strategy takes into account periods of grading inactivity. The velocity is then calculated as the sum of the duration attribute on each cluster divided by the number of evaluations created across all clusters in the set.
At the present time, we only externalize a simple representation of this data which we call velocity but there is the potential to draw timing graphs which visualize the periods of time a grader was active and inactive. We encourage feedback on how best to externalize the analytics information that we are collecting. Write to us at support@crowdmark.com with your feedback.
where the sum ranges over all clusters associated to a timeable object. For example, we sum over all clusters associated to a particular question or grader.
Implementation
Locating and updating a timeable’s cluster object
MAX_GAP = 10.minutes
DEFAULT_INTERVAL = 2.minutes
def self.touch(timeable, evaluation, stats={})
now = Time.now
cluster = timeable
.evaluation_timing_clusters
.find_by('ended_at > ?', now - MAX_GAP)
if cluster.present?
evaluations_count = cluster.count
evaluations_count += 1 if stats[:increment_evaluations]
cluster.with_lock do
cluster.update_attributes(
ending_evaluation: evaluation,
count: evaluations_count,
ended_at: now,
duration: now - cluster.started_at
)
end
else
timeable.evaluation_timing_clusters.create!(
starting_evaluation: evaluation,
ending_evaluation: evaluation,
count: 1,
started_at: now - DEFAULT_INTERVAL,
ended_at: now,
duration: DEFAULT_INTERVAL
)
end
end
Computing velocity
def self.velocity(timeable)
total = total_duration(timeable).to_f
duration = (total / 3600)
count = timeable.evaluation_timing_clusters.sum(:count)
return 0 unless duration > 0
(count / duration).to_i
end