Need help getting ready for finals? Click here to check out our 2024 Training Webinar series.

Share

Grading activity timing in Crowdmark

Crowdmark provides real-time insight into grading activity. Real-time means that at all times, the data displayed in the assessment dashboard is current. We track a velocity (evaluations per hour) metric related to questions as a whole and also for each grader within a particular assessment.

Question timing data on the Dashboard
Grader timing data on the Team page

How it Works

In order to support real time analytics, Crowdmark stores computed grading activity over intervals of time in a data object we call an evaluation timing cluster. We chose that name because we are analyzing what happens while the grading team performs “evaluations” of student work. We coalesce and store the data measuring grading activity over a period of time into “clusters”.

When a grading action happens such as drawing a freeform annotation, adding a comment, or assigning a numerical grade there is a side effect which we call a touch. A touch event takes in the thing in the assessment that we’re timing (a question, or a grader’s membership in an exam) and looks in the database to see if there is already a timing cluster started for that thing.

As time passes, new events can be added to a cluster. If a cluster is found for the current timeable thing, that cluster’s ended_at attribute is updated to the current time. This means that as grading actions occur, the cluster tracks a longer and longer time interval. The duration of the cluster is the difference between ended_at and started_at attributes of the cluster object.

If there was no cluster found for the timeable, a new one is created with a started_at attribute set to two minutes in the past and it’s ended_at set to the current time. This is because we make the assumption that the grader spends approximately two minutes on the question grading it before actually taking a grading action. The arbitrary selection of the two minute threshold appears to be effective but is not yet optimized.

Clusters can stop growing in time. A timing cluster can expire. That is to say, if a grader stops grading for a period longer than MAX_GAP (ten minutes), the cluster is no longer extended and a new one would be created. This strategy takes into account periods of grading inactivity. The velocity is then calculated as the sum of the duration attribute on each cluster divided by the number of evaluations created across all clusters in the set.

At the present time, we only externalize a simple representation of this data which we call velocity but there is the potential to draw timing graphs which visualize the periods of time a grader was active and inactive. We encourage feedback on how best to externalize the analytics information that we are collecting. Write to us at support@crowdmark.com with your feedback.

where the sum ranges over all clusters associated to a timeable object. For example, we sum over all clusters associated to a particular question or grader.

Implementation

Locating and updating a timeable’s cluster object

MAX_GAP = 10.minutes
DEFAULT_INTERVAL = 2.minutes

def self.touch(timeable, evaluation, stats={})
  now = Time.now
  cluster = timeable
    .evaluation_timing_clusters
    .find_by('ended_at > ?', now - MAX_GAP)
  if cluster.present?
    evaluations_count = cluster.count
    evaluations_count += 1 if stats[:increment_evaluations]
    cluster.with_lock do
      cluster.update_attributes(
        ending_evaluation: evaluation,
        count: evaluations_count,
        ended_at: now,
        duration: now - cluster.started_at
      )
    end
  else
    timeable.evaluation_timing_clusters.create!(
      starting_evaluation: evaluation,
      ending_evaluation: evaluation,
      count: 1,
      started_at: now - DEFAULT_INTERVAL,
      ended_at: now,
      duration: DEFAULT_INTERVAL
    )
  end
end

Computing velocity

def self.velocity(timeable)
  total = total_duration(timeable).to_f
  duration = (total / 3600)
  count = timeable.evaluation_timing_clusters.sum(:count)
  return 0 unless duration > 0
  (count / duration).to_i
end

Interested in learning more about Crowdmark? Get in touch for a free trial:

About Crowdmark

Crowdmark is the world’s premiere online grading and analytics platform, allowing educators to evaluate student assessments more effectively and securely than ever before. On average, educators experience up to a 75% productivity gain, providing students with prompt and formative feedback. This significantly enriches the learning and teaching experience for students and educators by transforming assessment into a dialogue for improvement.