Crowdmark provides real-time insight into grading activity. Real-time means that at all times, the data displayed in the assessment dashboard is current. We track a velocity (evaluations per hour) metric related to questions as a whole and also for each grader within a particular assessment.

Question timing data table Question timing data on the Dashboard

Grader timing data table Grader timing data on the Team page

How it Works

In order to support real time analytics, Crowdmark stores computed grading activity over intervals of time in a data object we call an evaluation timing cluster. We chose that name because we are analyzing what happens while the grading team performs “evaluations” of student work. We coalesce and store the data measuring grading activity over a period of time into “clusters”.

When a grading action happens such as drawing a freeform annotation, adding a comment, or assigning a numerical grade there is a side effect which we call a touch. A touch event takes in the thing in the assessment that we’re timing (a question, or a grader’s membership in an exam) and looks in the database to see if there is already a timing cluster started for that thing.

As time passes, new events can be added to a cluster. If a cluster is found for the current timeable thing, that cluster’s ended_at attribute is updated to the current time. This means that as grading actions occur, the cluster tracks a longer and longer time interval. The duration of the cluster is the difference between ended_at and started_at attributes of the cluster object.

If there was no cluster found for the timeable, a new one is created with a started_at attribute set to two minutes in the past and it’s ended_at set to the current time. This is because we make the assumption that the grader spends approximately two minutes on the question grading it before actually taking a grading action. The arbitrary selection of the two minute threshold appears to be effective but is not yet optimized.

Clusters can stop growing in time. A timing cluster can expire. That is to say, if a grader stops grading for a period longer than MAX_GAP (ten minutes), the cluster is no longer extended and a new one would be created. This strategy takes into account periods of grading inactivity. The velocity is then calculated as the sum of the duration attribute on each cluster divided by the number of evaluations created across all clusters in the set.

At the present time, we only externalize a simple representation of this data which we call velocity but there is the potential to draw timing graphs which visualize the periods of time a grader was active and inactive. We encourage feedback on how best to externalize the analytics information that we are collecting. Write to us at support@crowdmark.com with your feedback.

$velocity = \frac{\sum_{i=0}^n clusters[i].count}{\sum_{i=0}^n clusters[i].duration}$ where the sum ranges over all clusters associated to a timeable object. For example, we sum over all clusters associated to a particular question or grader.

Implementation

Locating and updating a timeable’s cluster object

MAX_GAP = 10.minutes
DEFAULT_INTERVAL = 2.minutes

def self.touch(timeable, evaluation, stats={})
  now = Time.now
  cluster = timeable
    .evaluation_timing_clusters
    .find_by('ended_at > ?', now - MAX_GAP)
  if cluster.present?
    evaluations_count = cluster.count
    evaluations_count += 1 if stats[:increment_evaluations]
    cluster.with_lock do
      cluster.update_attributes(
        ending_evaluation: evaluation,
        count: evaluations_count,
        ended_at: now,
        duration: now - cluster.started_at
      )
    end
  else
    timeable.evaluation_timing_clusters.create!(
      starting_evaluation: evaluation,
      ending_evaluation: evaluation,
      count: 1,
      started_at: now - DEFAULT_INTERVAL,
      ended_at: now,
      duration: DEFAULT_INTERVAL
    )
  end
end

Computing velocity

def self.velocity(timeable)
  total = total_duration(timeable).to_f
  duration = (total / 3600)
  count = timeable.evaluation_timing_clusters.sum(:count)
  return 0 unless duration > 0
  (count / duration).to_i
end