Unlike other platforms that you may have tried, Qualified does not score our code challenges by reading STDIN and matching strings printed to STDOUT. 


STDIN and STDOUT is how a program gets data in and out of it. When a developer tries to solve a challenge, if they are using a platform that utilizes STDIN and STDOUT, it means they are writing their code in a way that it needs to receive data via STDIN, and then print that data via STDOUT. The platform then tests the program by matching the string that was outputted from the program (via STDOUT) to an expected string that is coded into the test that scores the solution.

Doing things this way has the following issues:

  • It's awkward. Developers don't typically write programs this way. It doesn't feel like a real coding experience, it feels like you are being tested in a fake environment.
  • The input format can be incredibly confusing. It requires parsing multiple lines of text in a way that is alien to developers.
  • It results in code that does not look like what production code would look like, because you are writing to STDOUT instead of doing what you normally would do, which is to return a value from some sort of function/procedure.
  • It can be difficult to debug, because the output of the program is being used both for outputting the answer, and outputting debug data.
  • It is very limited in what you can test. You can only test what the program returns as string data and then matching that data to an expected string. There is no way to test whats happening inside of the program, and there is no way to handle dynamic data that may not always return a value that can be hardcoded (for example, time data).
  • Developers cannot write their own code tests in a way that is natural to them. Instead they are forced to send in convoluted string input formats. There is no way for them to writes tests that check for things like null values or invalid data types. These things matter, a lot, too programming. The fact that they are not supported is a big deal.

This is part of the reason why other online coding assessment platforms tend to focus on computer science algorithms. STDIN/STDOUT based testing limits what you can test, and algorithm challenges tend to be the only good fit. Other types of challenges can be very difficult, if not impossible, to achieve using this restrictive form of testing.

OK, so that's how other platforms score challenges. How does Qualified do it?

Qualified uses the same unit testing frameworks that developers already use when writing production code. When a developer is writing production code for a real product, they need to write code that verifies that their application code performs as expected. This is how they maintain high code quality and ensure bugs are minimized later on as things change. Typical large codebases can have thousands of tests in them. We figured, if these unit tests are good enough to test production code, they should be good enough to score a coding challenge, right?

There is a reason why other platforms don't use this format. It's a lot more work to implement. Qualified has put a lot of effort into implementing and maintaining support for dozens of different unit testing frameworks. Many of our languages even support multiple frameworks, in case someone has a preference.

Real-world unit testing provides the following benefits:

  • No awkward code. Developers write code exactly as they would for production code. In fact, in many cases you can copy production code and tests into one of our challenges and run it exactly as it is. 
  • Clean problem setup. Most challenges in our system typically setup a predefined method with its arguments already defined. All a developer needs to do is complete the function by returning the correct value given the arguments. This is a lot more clear and realistic.
  • Easier debugging. Instead of writing both the expected answer, and debugging data, to the same STDOUT, developers can simply write debug statements to STDOUT only, which is what they are used to doing in their own code.
  • Test anything you want. Since unit tests run inside of the program, you can test for things like data types, null values, existence of objects in memory, dynamic values, randomized testing, etc. The unit tests themselves are executable code, so they can do anything that you program them to do. 
  • No longer stuck giving algorithm challenges. Algorithms are great, but in all honesty, they don't actually get written that often in day to day development. Software engineering involves a lot of different types of programming tasks, and with unit tests, you can test them all. Anything that is tested in production code using unit tests, can be tested within our system.
  • Test Driven Development. Developers who are used to writing unit tests for their own code will feel right at home with our platform. They can write tests to guide them along their way as they try to complete the challenge. You can even require them to write their own tests, and review how well they wrote them to complete the task!
  • Control the display of the output. Our unit tests have the ability to output HTML, which can be rendered within the test results window. You can even output canvas objects. This output can be controlled by the unit tests that are scoring the challenge. This allows for incredibly versatile options for how you display test output. 
Did this answer your question?