Overview

Project Code Challenges is an advanced code challenge format. Unlike basic code challenges (the existing format you are probably already used to), Project challenges allow you to build a complex challenge structure, with multiple files, that allow you to mimic a project structure and more. 

Key Benefits

When compared to classic code challenges, project code challenges have a number of added benefits. They are far more powerful than simply allowing candidates and students to edit more than one file.

Multiple files With Advanced Security Options

Project code challenges not only allow you to freely define files, allowing you to create a complex challenge structure, but they also allow you to configure a number of advanced options for each file and folder, allowing you to control how much access a developer has to the challenge structure. For example, you can configure certain files or directories to be hidden, so that the developer doesn’t know they are there, or you can configure them to be readonly, so that the contents can be viewed but the developer has to solve around them instead of editing them directly. You can also configure certain files to only be included on submission, to be only included when developer tests are ran, or both.

These options are incredibly powerful, as you can configure a challenge with all sorts of constraint possibilities, allowing you to focus the challenge exactly how you want it.

Submission Results Are Always Hidden

With our classic code challenges, developers were able to submit their code as many times as they wanted to. This resulted in them being able to see their final score, and to be able to continue working until they completed everything. 

With project challenges, submission results are no longer shown. This was done because the flexibility of project challenges allows for more configuration possibilities. If you want to show each developer their test results without showing them the actual tests, you can simply create hidden test files that run with the developer’s tests. If you want certain tests to only run on submission, hidden from the developer’s view, you can much more easily achieve this now that submission results are not shown. 

Advanced IDE Configuration

Project challenges have a more advanced tabbing system, that allows developers to drag and drop files and special tabs to suite their desired configuration. This feature will eventually be supported with classic code challenges as well, but for now, it’s a project challenges only feature.

Key Disadvantages

While project code challenges provide a lot of power, they also have some disadvantages compared to classic code challenges that you may want to consider when choosing this challenge type.

Steeper Learning Curve

For challenge authors, project challenges have a steeper learning curve. They provide an advanced array of options for configuring a challenge, but this may be initially overwhelming if you are trying to get started with a simpler style of challenge.

Single Language Only

Unlike classic code challenges, project challenges only support a single language. This means that when adding a project challenge to an assessment, the candidate (or student) will be forced to use the language that the challenge is built-in. As such, project challenges also do not currently support a language generator feature like classic code challenges do.

Limited Review Features

For a temporary time, until some features are completed, project challenges will be limited in the features that you can use when reviewing submitted solutions. Key features that are planned but not currently supported for project challenges are:

  • Code Comparison
  • Code Similarity
  • Limited Language Support

We do not currently support all of our languages within project challenges, and for some, such as SQL, it may never make sense for us to do so.

Content Possibilities

With the project code challenge format, Qualified is able to better support additional testing scenarios. This is due to the ability for the project structure to be customized so well. For example, you can do some of the following things:

Assess Communication Skills

Developing is more than just writing code. It involves many other aspects, such as collaboration and communication. Developers must do things like write descriptive pull requests, respond to emails, provide feedback and recommendations via email and chat, write documentation, respond to customers, and more. 

Since project challenges allow you to define multiple files as part of the challenge, you can define certain files that are meant to represent certain types of communication. For example, you could define an “email-response.txt” file and ask candidates to fill that file out with the response they would send someone after looking at their code. Here are a few more ideas of how you can use files for non-coding tasks (list is not exhaustive):

  • Provide a “pull request” file, and ask developers to not only complete the solution, but to add their pull request title and description, as if they were checking their solution in.
  • Write up a status report email or Slack message
  • Write supporting documentation for their newly added feature or API

We should note, that you can optionally mark these “communication” files to be included on submission, which would allow you to write submission test cases that check for their existence. You could parse the files and incorporate any sort of logic you want to try to develop a set of automated scoring around the communication. This is of course in addition to using our review feature to leave subjective feedback and ratings.

Provide Additional Documentation

Every challenge in Qualified includes the ability to use markdown to write instructions. However you may want to provide additional details, such as documentation, that developers can use when taking the challenge. You are no longer limited to cramming everything into the instructions markdown. For example, you could provide a “docs” folder and include text or markdown files within it, marked as readonly. This would allow the candidate to view additional resources without having to deal with a massive set of instructions.

You can provide entire directories of documentation, or supporting materials to go with the challenge.

Forced Code Restrictions

The ability to restrict what files a developer can see and edit is a powerful tool. For example, you could simulate a situation where a developer must patch a 3rd party package that they don’t have direct access to, or you could simulate a developer working with two components of an application, where one of those components is handled by a different team and the developer is unable to make changes. This would force the developer to find workarounds, and make decisions that achieve the task while demonstrating if they can do so without implementing a poorly designed hack in the process.

Improved Test Fixtures

Often times when writing tests, we have found there are times where a lot of code is being embedded within the test files. For example, when mocking an API response or an expected JSON output. At the same time, these tests are not easy to read for developers taking the challenge, and hard for challenge authors to maintain. With read-only and hidden files, you now have the flexibility of extracting test fixture data out of your test files, where you can choose to allow developers to see those fixtures, but not allow them to change them. 

You could create a “fixtures” directory and utilize JSON, YAML, or any other type of text file to define your test fixture data.

Refactoring and Bug Fixing Challenges

Not exactly newly introduced, since we already support this in a small format with basic coding challenges. However with project challenges, testing a developer’s ability to digest a reasonably large codebase and determine what should be refactored, or what should be fixed, opens up a much wider set of possibilities for targeting these types of programming tasks.

These are just a few of the many possibilities we think are possible. The creative possibilities are numerous and we are excited to explore these with our customers further.

Did this answer your question?