In CPSC 110 we use an automatic grader (autograder) to grade lecture work, labs, problem sets and exams. The autograder score is never the only way an assignment is graded, and what percentage of the score it represents depends on the kind of assignment:
The auto-grader cannot grade your file unless it passes the Check Syntax button in your Dr Racket. So before you press handin always press Check Syntax. If Check Syntax shows errors you MUST fix those before you handin, otherwise you will get a 0 for your autograder score. Depending on the assessment your total score may also be 0. So Check Syntax often as you work.
The Run button is strictly more demanding than the Check Syntax button, so you may often press Run instead. If your file runs, even if some tests fail, or some specific tests have errors, then it can be graded. If you run your file and there are problems, then be sure to press Check Syntax before you handin so you know whether those problems will prevent it from being graded.
This section describes the feedback you will receive from the autograder for HtDF problems.
The signature item is the most straightforward. It says whether the signature argument and result types are correct.
The item is evaluating the submitted check-expects only. The score indicates whether your tests are valid - meaning do they match the course staff's interpretation of the problem statement. All the arguments must conform to the function's signature; and the result value must be correct for those arguments.
The test validity score is very useful when you start an HtDF problem to be sure you understand the problem statement. You can write examples wrapped in check-expects and submit them. If they are valid you know you can use them to design the function. If they are not valid then you need to reconsider the problem statement. To encourage you to do this, the autograder will produce test validity scores even if you resubmit your file before the cooldown period. Take advantage of this to make sure you know the details of what the function should do before you move on to the template origin, template and coding the function body.
Note that submitted tests can pass but still not be valid. If, for example, the submitted tests pass, but they are not valid and additional tests fail, then this is a good sign that your interpretation of the problem statement is incorrect.
This item is evaluating the submitted check-expects only. This item evaluates whether the submitted tests, as a group, cover all the argument values they should, including boundary conditions and other particular argument values that should be tested.
Note that submitted tests can pass and be valid, but still not have complete test argument coverage. If for example there is no test for a boundary condition in the function then this item will be incorrect.
This item is evaluating the submitted check-expects only. For this evaluation the autograder has a set of its own faulty versions of the functions. These are versions of the function that don't quite work properly. For each of those faulty functions the autograder runs all of the submitted tests. If one or more of the tests fail that's good. But if no tests fail for one of the buggy functions then it means that fault was not detected.
This item is incorrect when one or more known faults are not detected. As with test argument coverage, submitted tests can pass and be valid, but still not detect faulty functions
This item is evaluating the (@template-origin ..) tag. The tag should appear in the htdf function design, before both the function definition and the @template tag. The grader checks to see that all correct template origins are listed, and that no incorrect origins are listed.
This item is marked against the template origin tag that the course staff believes is correct for the problem.
This item is evaluating the (@template ..) tag. The tag should appear in the htdf function design, immediately after the @template-origin tag and before the function definition. The grader checks to see that the function definition template in the tag is correct.
This item is marked against the template origin tag that the course staff believes is correct for the problem.
This item is evaluating the function definition as compared to the template origin and/or template tags. Among other things this assesses whether the rules about editing cond questions, and question/answer pair order are followed in the actual function definition.
This item is marked against the template origin tag and/or template tag that the course staff believes is correct for the problem.
This item is evaluating whether the submitted tests run and pass.
This item is evaluating whether the course staff's additional tests run and pass. These are tests which check whether the submitted function has the behaviour that the staff wants it to have.
A common situation is for submitted tests to pass but one or more additional tests to fail. This indicates a situation where the submitted problem set has a self-consistent design; but not quite for the right problem. In other words, your function does what you want it to do, but it doesn't do what the staff wants it to do.