2024-03-13-NotebookRubric-Reference

	A	B	C	D	E	F
1	Jupyter Notebooks for Teaching and Learning Rubric, Cécile Hardebolle and Aditi Kothiyal, CC BY 4.0 International
2	#	Criterion	All aspects considered	Aspects partially considered	None of the aspects considered	Comment

3		Structure
4	1	Learning objectives	What students will learn is made explicit at the beginning of the notebook in the form of objectives that describe what students will be able to do after completing the notebook. The objectives are formulated using action verbs and describe outcomes in a measurable way (they could easily be translated into assessment questions for instance). Example: Students will be able to carry out hypothesis testing using R.	The notebook includes some overall goals allowing the learner to understand what the notebook is about, but the goals are not measurable or student centered. Example: We will explore various ways to implement a system that changes the voice of a speaker.	Not present
5	2	Segmentation	The content is segmented into a sequence of distinct smaller sub-topics with an explicit overall organization, visually separated into sections. Example: Steps in solving a problem are made clear by different headings in Markdown or HTML.	The content is segmented and sequenced, but the size of the chunks is not appropriate (too large / too small) or the organization is unclear.	Content not segmented	This criteria is about overall textual content, not code. There's a separate criteria regarding the structure of code.
6		Disciplinary thinking
7	3	Disciplinary embedding	The activities, examples or datasets correspond to representative tasks in the discipline of the students (e.g. from the students' study domain). Example: a notebook for material science students illustrates the use of machine learning techniques to predict the properties of a material.	The activities, examples or datasets come from varied domains, are not necessarily consistent or related to the discipline of the students. Example: a notebook for chemistry students illustrates clustering techniques on a dataset about pinguins.	The activities, examples or datasets are abstract (not representing anything in particular). Example: applying the gradient descent technique on a set of numbers.
8	4	Experts thinking modeling in the discipline	The notebook draws students' attention to conceptual or methodological reasoning in the considered discipline. Ways to investigate or think about problems or concepts are explicit, presented with heuristics/steps and illustrations or examples as well as explanations as to the rationale behing it. Example: A notebook for a data analysis lab explains and has students follow the steps of the inquiry cycle (formulating a question, stating hypotheses, analysing data and testing hypotheses, concluding based on evidence) and asks them to summarize the steps at the end.	A methodology to investigate or think about problems or concepts is used but not presented explicitly or the rationale behind the steps in the reasoning is not presented.	No methodology is presented.
9		Computational thinking
10	5	Experts thinking modeling about computational problem solving	The notebook draws students' attention to the computational/execution aspects related to the contents. Example: A notebook presenting code includes explanation about the ways in which the code is structured and why (for instance using functions to be able to reuse code), discusses issues related to execution time or limits of computational methods, or has students practice specific computational skills such as debugging or interpreting errors.	Issues or questions about computational/execution aspects are sometimes mentioned but students' attention is not explicitely drawn to it.	No mentions about computational aspects.
11	6	Code visibility	There is an alignment between the teaching/learning purpose and the visibility of the code. For notebooks with the purpose of learning programming, the code (or parts of the code) that is illustrating concepts to be learned is made visible and explained. For notebooks without programming goals, code is hidden either partially or totally. Example: for a demonstration notebook made available to students with activities to explore the interactive visualization, the code of the interactive visualization is hidden in an external python file.	The type of code that is shown or hidden is not necessary well aligned with the teaching/learning goals. Example: A notebook where students practice performing t-tests on a dataset includes a very long piece of code generating a complex visualization showing statistical distibutions, instead of hiding it in an external python file (and keeping only the code necessary to perform the t-test).	There is no alignment.
12	7	Code readability	If code is visible, the code is clearly structured and systematically presented with comments (in code) and/or explanations (in markdown) that add information to the code in order to help students understand the code. Examples: - A function that students have to complete includes a description of the role of its parameters including what they represent, examples of values and their units. - The line of code "pg.ttest(data['x'], data['y'])" is commented to indicate what the parameters of the function are, how the values were chosen and how to interpret the expected result.	If code is visible, the code is presented with some comments or explanations but not systematically or not providing complementary information. Example: The line of code "pg.ttest(data['x'], data['y'])" is commented with "Call to the t-test function"	No comments or explanations	NA if the code is not visible
13		Supporting representations
14	8	Choice of representations	In addition to text and code, the notebook includes additional representations, whether static (e.g. equations, tables, graphs and images) or dynamic (code-generated), to illustrate important concepts or processes students are supposed to learn. If existing, the representations follow standards from the discipline. Example: A notebook about a classical mechanics problem includes a free body diagram.	The chosen representations do not necessarily focus on important concepts or processes, or do not follow standards from the discipline.	No representations other than text and code.
15	9	Design of representations	The representations include cues that help students identify the important elements (signaling). Related information are grouped (spatial/ temporal contiguity). There are explicit links between multiple representations (e.g. math equations, code, visualization, diagram) such as pivot elements or dynalinking. Example: The names of variables are the same in math equations and in python code.	Important elements are difficult to identify or related elements are not grouped. Links between multiple representations are not systematically implemented throughout the notebook.	No signaling, contiguity or links between representations.	NA if there are no multiple representations (e.g. only text or only code).
16		Learning activities
17	10	Activities	The notebook contains some explicit questions or activities (quiz, open question, mini-exercise...) requiring students to actively process information at key points in the contents. Example: An interactive textbook includes a series of short open questions at the end of a section for students to review the key points of the section or an interactive quiz allowing them to check their understanding.	The notebooks contains sparse questions/activities or the questions/activies are separated from the contents.	No activities or questions engaging students in thinking throughout the notebook
18	11	Progression in activities	There is a progression in the learning activities included in the notebook, taking into account students' prior abilities and the teaching/learning purpose of the notebook. Example: a notebook that involves programming for novice students starts with code reading activities or worked examples and includes code modification or completion activities before tasks where students have to write code from scratch.	There is a partial progression, for instance not taking into account students' prior knowledge.	There is no progression.	NA if there are no activities.
19	12	Feedback to students	Beyond basic code execution result, the notebook presents students with feedback in a way that is actionable (i.e. provides information beyond correct/incorrect) and adapted to students' activity (i.e. personalized). In the case of assessment, students receive comments in addition to the grade after correction. Examples: - A notebook with programming exercises embeds a set of automated tests ("sanity checks") with specific feedback messages that students can run to check their code. - An interactive textbook includes a series of interactive quizzes with feedback messages that allow students to check their understanding.	Some feedback is presented to students (beyond code execution result) but the feedback is not actionable (no information beyond correct/incorrect or grade) or is general (e.g. predetermined solution)	No feedback presented to students	NA if there are no activities.
20	13	Feedback to teachers	The teacher has access to information about students' learning process in terms of completion/progress and perfomance on different tasks. Example: an exercise worksheet includes embedded interactive polls at the end of the main sections that allow the teacher to monitor students' progress as well as the correctness of their results	The teacher has access to opinions from students on the notebook but not to direct information about student learning.	No way for the teacher to get feedback.	NA if there are no activities. Assessment type notebooks are considered as providing feedback to teachers on students' learning.
21	14	Reflection questions	The notebook contains questions engaging students in self-reflection, leading them to step back and review what and how they have learned. Example: an interactive textbooks includes an open question "What did you learn from this notebook?". A notebook for an open-ended project includes an open question "List for yourself the different steps involved in analysing a dataset based on what you have done in this notebook.".	The notebook contains some questions leading students to deepen their understanding of the contents or thinking critically about results but not necessarily to think about what or how they learned Example: a notebook with statistics exercises include the question "What are your thoughts on this regression fit?"	No reflection questions in the notebook
22		Assessment
23	15	Grading scheme	A grading scheme is present with points for the different tasks / criterias and includes a description of the expectations or the different levels of performance. Examples: A programming exercise where students have to write a function and graded over a total of 3 possible points comes with the following grading scheme: - 1 point: The function computes the expected result - 1 point: At least one edge case is addressed - 1 point: A documentation is included in the form of a docstring with a description of parameters and return A notebook serving as a template for a lab report includes a grading scheme with several criteria and associated levels of performance such as: Provides interpretation of statistical analysis based on hypothesis, including explanation of relevant values. - 3 points: Statistical analysis is clearly interpreted and conclusions are properly explained. - 2 points: Interpretation of statistical analysis and conclusions reveal gaps in understanding - 1 points: Interpretation of statistical analysis and conclusions are invalid. - 0 points: Interpretation of statistical analysis and conclusions are incomplete or missing.	A grading scheme is present but the description of expectations / levels of performance for the different tasks/criteria is missing.	Grading scheme not present	Only for graded assignments.
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100