The NCLab Project SANDI Computational Thinking assessment is a 16-step series of tutorials, tasks, and quizzes that inform us about a candidate’s ability to:
The method used in the assessment is based on one developed at Stanford University around 1980 to teach “computational thinking” to engineering students. Educators quickly realized that the method worked well with children and non-technical people and have used it in many courses over the decades. NCLab has developed the most sophisticated version of the method, described in the next section.
Computational thinking is a term coined in the 1950s to describe how machines think: in other words, what inputs, procedures, and outputs machines need to successfully and reliably carry out a task. As humans, we have a lifetime of experiences that guide how our brains filter information and make decisions. Even sophisticated machines can’t match what we do. Computational thinking builds a bridge.
Nearly all jobs and professions interact with some automated process. This assessment gives us an idea of a candidate’s readiness to work in the modern technological environment.
Since introducing the method in 2013, NCLab has monitored how people learn with it and made many refinements over the years. NCLab developed an assessment based on the method to predict how people would perform in NCLab’s complete training programs and has found it to be an effective screening tool. For Project SANDI, since all current jobs interact with automation and technology, this assessment provides valuable information for a broad range of career outcomes.
Working with Karel is like learning how to play chess. A six-year-old can learn the basics, but could still be working on new moves at age ninety. The method is elegantly simple.
A robot named Karel (a nod to the author who coined the word “robot” in the 1920s) navigates a grid, detecting various objects. The objects are either collectibles, containers, or obstacles. Karel has sensors that detect the type of object. Karel only has five commands: go, left, right, get, and put.
We use these commands to write procedures for the robot. We can test the procedures as many times as needed to make sure they work. This “iterative” testing procedure is engaging: the results are immediate and people use the feedback to verify, debug, or improve the procedure before submitting it for grading.
This is not just another video tutorial with questions and answers. This is an interactive dive into working with automation.
There are 16 levels to the assessment, which test four skill areas in computational thinking. Within each skill area, the candidates complete four levels:
Level x.1 Read a tutorial and play an interactive demonstration.
Level x.2 Practice the skill by completing exercises to solve the first graded task.
Exercises include hints, templates, and solutions.
Graded task #1.
The graded tasks behave like exercises but do not include the solution.
Level x.3 Graded task #2.
Level x.4 Quiz based on skills learned so far.
This quiz tests critical thinking rather than simply testing recall.
What does a candidate learn about and demonstrate in each of the four skill areas? In the assessment, the commands are limited to what the robot can do. This might seem a little abstract.
To understand each skill area, it helps to think of real-life analogies. Here are some examples from a school cafeteria lunch line.
Assessment Skill Area
|Skill Area Description||Everyday Analogous Example
(Cafeteria at lunch time)
|Skill Area 1 (Levels 1.1, 1.2, 1.3, 1.4)||Learn about and write simple procedures.||Make peanut butter toast:
1. Put a slice of toast on a plate.
2. Scoop out 2 tablespoons of peanut butter from the jar.
3. Spread the peanut butter on the toast.
|Skill Area 2 (Levels 2.1, 2.2, 2.3, 2.4)||Learn about and write procedures that involve repeating patterns.||Prepare 12 peanut butter toasts.
Repeat instructions 1 to 3 for each order.
|Skill Area 3 (Levels 3.1, 3.2, 3.3, 3.4)||Learn about and write procedures with unknown conditions.||If a child requests cinnamon, sprinkle cinnamon on top of the toast.|
|Skill Area 4(Levels 4.1, 4.2, 4.3, 4.4)||Learn about and write procedures with variable controls for repetition.||Prepare peanut butter toasts for everyone in the lunch line*. We don’t know in advance how many kids are getting school lunch today.
*Not much choice at this school and no one is allergic to peanut butter or toast!
If you created a persona in Module 1, think of a real-life scenario where they might use these skills.
In general, this is how automated procedures work. A candidate might work with such processes in manufacturing assembly lines, data processing, medical procedures, retail ordering, or farming.
The candidate needs a computer with an up-to-date browser (Edge, Chrome, Firefox, Safari for example) connected to the internet. It cannot be done on a phone. The assessment can be taken at a library, a school or training facility, or at home. All the work is saved on the cloud, so the physical location does not matter.
Assessment Code for Explorers: dbcrs
The candidate goes to the NCLab SANDI page
The instructions for setting up and taking the assessment are on the page. Briefly, the candidate will follow these steps:
Fill out and submit a form to create an NCLab account with the first and last name, email address, and assessment code.
Open the email containing the activation link. Click on the link to load the NCLab desktop page.
Start working on the assessment.
NCLab recommends setting aside three hours for quiet, uninterrupted work. It is best to do the assessment in one sitting. If the candidate needs to stop, they can log off and resume progress later. Their progress is automatically saved.
At the end of the assessment, the candidate schedules a call with an NCLab Intake Specialist from a form that shows available dates and times. This 15-minute interview confirms the candidate’s location and stage in the process and asks three open questions about the assessment. The interview provides anecdotal information about the candidate and their experience with the assessment and verifies that they have taken it themselves.
The candidate will receive a summary report based on the completed online assessment and the interview.
The navigator will receive a full report based on the completed online assessment and the interview. If the navigator has sent a candidate to take the assessment and has not completed it, they can request a partial score.
NCLab uses a scoring system to rate performance on four main factors:
Perseverance and completion: The total weighted score can range from 0 to 100. Anyone completing the whole assessment will score a minimum of 60 points.
Attention to detail and self-monitoring: The candidate can test their procedure and make corrections before submitting it for grading. A candidate who reads the tutorial carefully does all the exercises, troubleshoots their answer, and checks it against the requirements before submitting it to the grader is more likely to pass the grade on the first attempt. A failed attempt still allows the candidate to go back and rework the answer and resubmit as many times as needed to pass.
Independent problem solving: Because NCLab supplies both hints and templates (fill in the blank answers), we can measure how independently the candidate solves each graded task. Using a hint for a task shows that they need some additional support. Using a template indicates that they depend on a specific outline to solve the task. If they can solve a task without either a hint or a template, that shows independence. Certain jobs may require following procedures with specific steps, whereas others require more independent problem-solving.
Critical thinking: At the end of each cycle, the quizzes require the candidate to think about each possible response rather than simply recalling information in the tutorials. They may, for example, look at different patterns and determine which one works best to solve the problem.
The summary report provides two aggregate scores, one for problem-solving and one for critical thinking. The problem-solving aggregate score is based on weighted scores on the subtests. The critical thinking score is based on the quiz average score.
The report also contains the answers to three questions posed during the post-assessment interview and a short summary of the call.
See sample assessment report below.
All Community of Practice members will work to identify and place eligible candidates into a program that leads to completing Step 10: Getting a Career!
To refresh your memory, here are the ten steps:
A candidate might arrive at the NCLab SANDI assessment via different avenues.
No matter how people reach https://nclab.com/sandi/, they have arrived at the right place to take the SANDI assessment.
As part of Step 3 or Step 4, the library navigator will point the candidate to the NCLab SANDI webpage. The library navigator will receive the report to assist in evaluation in Step 5 and will pass it along with other pertinent information to the preferred training provider. NCLab is excited to see the libraries as partners in this project. We have worked with libraries across the state for many years and are ready to coordinate their efforts to attract and screen candidates.
The college navigator may have candidates who have not taken the assessment but could benefit from it.
Since NCLab courses are listed on ETPL, some candidates may already have an interest in NCLab training packages and choose to take the assessment.
People interested in data analytics or programming may find NCLab independently. From an initial call, NCLab might determine that the person should be evaluated for eligibility under the SANDI grant. They may have already taken the assessment before applying for the grant funding.
Regardless of the source, NCLab will share the assessment results with the appropriate navigators in the SANDI program.
The assessment was initially designed to screen people for the data analytics and programming training programs at NCLab. SANDI organizers envisioned it as a screen for technology-related career training programs. However, there are good reasons to steer a broader group into the assessment.
People have hidden talents. Very few assessments test computational skills, and a person may not be aware that they have these abilities. That discovery can lead to a wide range of well-paying career opportunities.
Technology is not just for specialists. Everyone has to interact with automation. This assessment helps determine their readiness.
The assessment provides how an individual handles self-paced, online learning. This is a new paradigm for many people.
Taking the assessment can answer a straightforward question: Can a person learn new things and apply what they learned? In this sense, the assessment is one indicator of success for any program within SANDI.
The ACT WorkKeys National Career Readiness Certificate (ACT WorkKeys NCRC®) is an assessment-based credential issued at four levels; Platinum, Gold, Silver, and Bronze. The NCRC measures and certifies the essential work skills needed for success in jobs across industries and occupations. With an NCRC, you can: