This article quickly explains key concepts of the project challenge editor. For guidance on navigating the UI, you can also check out our interactive guide.
Project Code Challenges have a more advanced editor than Qualified's basic Code Challenges. This article will quickly guide you through the key concepts that you should understand.
The concepts we will cover are:
- Starting a new challenge
- Project Files vs. Solution Reference Files
- Candidate Tests vs. Submission Tests
- File Access Permissions & Submission Ignore Paths
Starting a New Challenge
To start building a new project code challenge, go to the challenges tab and click "Create a Challenge". A list of challenge templates will appear.
If your subscription has access to project code challenges, you should see a number of templates at the top of this dialog in which you can build a new challenge with. Pick the template that corresponds to the type of challenge you wish to build.
- Not all subscriptions contain access to the Project Code challenge type. If you don't have access to this challenge type, please contact Qualified's customer support team.
- Contact us if a template matching the technology stack you wish to build a challenge with does not exist. We can work with you to add support for the technology you intend to assess on.
Project Files vs. Solution Reference Files
When working with Project Code Challenges, you will be working with two different sets of files. You will be defining the initial state of the challenge, as well as a reference final state, so that you can build the challenge and test that it can be completed at the same time.
- Project files are the files that will be included within the challenge. Candidate's taking a challenge will start with these files.
- Solution reference files are contained in a separate file tree and are used to provide a working reference solution.
Why do we need a working reference solution?
A reference solution must be provided in order to demonstrate that 100% of the challenge's submission test cases can be completed. This ensures that you are building a passable challenge, and also makes it possible for you to easily build the challenge, while also defining it, and verifying that the tests make sense.
What happens when I have both a project file and a reference solution file?
When testing the reference solution, the two different sets of files get combined together. Any time you are running the reference solution, the file within the reference solution file tree will be used in place of the one that is included within the project files tree. In other words, your reference files will always overwrite the corresponding project files. Any project files that are not overwritten by the reference solution will still be included, as long as you have configured them to do so (more on that later).
How do I switch between working with the project files and reference solution files?
There is a toolbar, above the file tabs, which allow you to configure which fileset you are working with. Whenever you press one of the run buttons, it will run the fileset you have selected.
Candidate Tests vs. Submission Tests
When designing a project code challenge, you should keep in mind that a candidate will be able to run unit tests (Candidate Tests). The candidate can also submit their solution (Submission Tests). Unlike Qualified's traditional Basic Code Challenges, candidates only submit their solution once before moving on to the next challenge in their assessment, and are never shown the submission results. This is different from basic code challenges, where a candidate can submit over and over again and see the results.
How do I configure different tests to run for candidate tests vs. submission tests?
Every file within the project files tree can be configured to be included on candidate tests, submission tests, or both. If you want to include a test file that is meant to only be ran on submission, then you would mark that file as "include on submission" and uncheck the option to "include on candidate tests". Every project template you start with should already have a file like this pre-configured for you.
How do I prevent candidates from seeing the submission tests?
You would set the file access to
Hidden , which would prevent the candidate from seeing the file. More on this in the next section.
Managing File Permissions
The most powerful feature of project code challenge isn't just that it has multiple files, but the fact that you can manage various permissions for how candidates can access them. This allows for a lot of flexibility in how you control the coding and execution environment that a candidate is working with.
Every file within the project files tree can have one of four access permissions:
- Read Only
Read Only are pretty self explanatory.
Hidden are a bit more special. They both prevent a candidate from viewing the contents of a file.
Restricted still allows a candidate to see the file within the file tree, so that they are aware of its presence.
Hidden completely hides the file from the file tree.
These open up a lot of use-cases that may not be completely obvious at first. For example, you can use
Read Only files to prevent a candidate from editing a specific file, forcing them to work around the issue as a designed constraint of the challenge. You can also use them to provide things like test fixtures, so that you can see the data that is being tested, but cannot edit it.
The most useful case for
Restricted files is to include test cases that you don't want to show candidates, but you can also use it to include helper code that you don't want to overload the candidate with, or provide test fixtures that you don't want to keep a secret. There are many more possibilities.
What I want to allow a candidate to modify submission tests, but I don't want those changes to be included when the solution is submitted?
You wouldn't want a candidate to be able to affect the code that is used to score the challenge. For this reason, all project code challenge templates are configured so that all test file paths ignore candidate changes to those files.
For example, for NodeJS + Mocha projects, only files below the
/test/ folder are included as test files. This means if you want, you can allow candidate's to edit any of those test files, for their own development efforts, and allow them to run those tests as candidate tests. However when they submit their solution, their changes will not be included, preserving the integrity of your original submission tests. You will still be able to see their edits though, when reviewing their submitted solution.