Thursday, December 15, 2011

Open-Internet Tests

Back in December 2010  I was was issued a challenge: "If all your tests were open-internet tests, how would you have to write them differently?"  This question has been vexing me ever since that time, and when I ask it to my colleagues I am often greeted with blank stares.

Jump forward to July 2011.  While at Alan November's conference in Boston, MA I had the opportunity to hear Stephen Wolfram give a presentation on http://wolframalpha.com in which he demonstrated how most basic recall and computational questions could be answered in seconds using the site.  At that point I knew I had to open up my Chemistry exams to the internet.

August 2011 rolled around and on the first day of school I explained to my students that all my tests would be open-internet tests.  I did not change them in any way from the previous year, I just wanted to find out how students would do on the tests compared to previous years.

It is now the last week of the semester and I have made a few observations.  I have not yet compared any test scores to previous years, but plan to do so once all the final exams have been taken and the grades are in.  However, I have noticed a few very interesting trends.  Four categories of students seem to have emerged.

1. The Purists:
2. The Opportunists:
3. The Google Challenged:
4. The Disinterested:

The Purists
These are the students who neither want nor need the additional resources available on the internet to succeed in my class.  They choose to learn how to solve the problems and understand the concepts in the same manner that students in previous years have.  I teach it, they internalize it, and can use it without other resources.  Occasionally these students will make careless errors on exams, and when asked why they didn't use the internet, the typically respond with "I don't want to, I want to be able to do it myself."

One of the objections I hear from my colleagues when I tell them I have open-internet tests is that I am helping to create a generation of people who are reliant on the internet, and soon we will have no one who can perform basic math because we will all be dependent upon calculators the internet.  I am happy to report that regardless of the availability of "cheats," some students will still want to dig in and learn all there is to learn on their own and will desire to be able to do it themselves.

The Opportunists
These students are the ones who couldn't thank me enough for opening up the internet during exams.  This group is composed of students who struggle with recall, but can perform quite well with a simple list of equations or notes to jog their memories.  Students from this group generally have found one or two resources (i.e. wolframalpha) that they have learned to use effectively to help them with the parts of the problems that require some of the lower end of Bloom's taxonomy so they can concentrate on the upper end.  I am confident that these students have benefitted most from the open-internet tests.

The Google Challenged
These students try to use the internet to help them on their tests, but simply are horrible at searching. Often times they will type in my test question in its entirety into Google and expect it to give them the answer.  They want to use the resources available, but do not know how.  For these students, I plan to spend additional time teaching them the skill of developing a strategic search strategy, generating an effective search term, searching the internet, filtering the results, and reporting those results accurately.  I find that this group is made up of the same students who work hard, but continue to struggle to put all the pieces together in other classes I have taught.  My class seems to lack the necessary relevance to help this group connect all the dots.

The Disinterested
This is the most intriguing group of students.  I call this group The Disinterested because they seem to lack the interest in the class/content/school/whatever to even bother to use the resources available to them. I have noticed that when reviewing a test with a student from this group that very simple questions with very simple answers that could be easily obtained from a very basic Google search have been missed.  When I inquire if they used the internet as a resource, the typical response is "no, I didn't feel like looking for the answer."  Clearly my exam does not accurately reflect what they can or cannot do, it simply reflects what they did or did not do.  Big difference, with loads of implications to the validity of testing in general, but I'll save that for a different post.

The letter grade breakdown:
The Purists have As
The Opportunists have As or Bs
The Google Challenged have Cs and Ds
The Disinterested have Ds or Fs

To be honest, these grades do not surprise me in the least bit.  I'm pretty sure that when I look at the grades compared to last year that I will have a few more As and Bs than before because The Opportunists have capitalized on the available resource and have improved because of it.  Everyone else...business as usual.


The Wild Cards
There is one additional group of students not addressed above.  They are the students who opted out of my tests and chose to complete an alternate assignment that demonstrated their understanding of the objective in the unit instead.  I'm still working out some of the details of how I assess these students.  Currently, I check to make sure their project or product has addressed all the objectives and demonstrates their understanding, but it's still pretty subjective.  However, most students who have gone with this option have blown me away with the creativity and quality of their work.  I will also admit that some of my students are on to the fact that things are a little squishy with these assessments and are choosing this option as an easy way out. 


Conclusions and reflections:
1. By using the same exams as in the past and opening them up to the internet, a small portion of my students have been able to improve their grades.

2. I really need to change the way I assess my students.  The Purists, The Google Challenged, and The Disinterested have not benefitted from open-internet tests.  I need a better way to assess the understanding of the Google Challenged and the Disinterested, and I need to be able to explore the upper limits of The Purists.

3. My students continue to treat tests as a means to acquire the sufficient number of points to obtain the letter grade that will keep their parents off of their backs.  They do not treat them as the diagnostic tool needed to help me determine what a student knows and can do.

4. I'm tempted to dump my summative unit tests entirely for the second semester and require that all students create a project or product that demonstrates their understanding of the objectives in the course.  I also hope to incorporate more project driven units that are less content focused, and more context driven.

5. I need to amp up the quality control on the alternate assessment, especially if I only accept this form of assessment in the second semester.

I would appreciate any comments or feedback.

8 comments:

Sylvia said...

Thank you for sharing! These are really interesting reflections. Do you think you taught any differently this year knowing that the tests would be open Internet?

Anonymous said...

Very interesting. Could you post part of one of your tests so we might have more context about how students could use the internet to help them on the test?

Unknown said...

Great idea. I've often wondered about open-internet tests and usually saw it as a negative aspect. Since tests are meant to be an indicator of content learned , it seemed funny to me to allow them to have the internet as a resources.

I have to say, I am great at Googling and assumed that most students would be as well. Its interesting to know that many are not!

I'd like to know what format or form you use to hosts your tests.

With an increasing amount of technology becoming widely available to all students, I think that this is a good move. Maybe it would be another method of differentiating assessment strategies.

www.teachscienceright.com

Ben Arcuri said...

I have ben a chem teacher for the past 10 years, and dedicated most of that time to assessment practices and techniques. This idea has a lot of possibilities. I think it depends on what you feel the role of a test is...

Sorry if I missed this, but how much time do you give your students to write the test? Is it the same as last year? What if a student simple finds your answer key to a worksheet and models the test question after the review sheet? What does the test show you then?

Why use an internet based test...why not use the textbook? Having trouble seeing the difference.

Unknown said...

Students have one class period to complete the exam, just like they would in a traditional test situation.

If the student just models the answer off an online solution, then I need to ask better questions. This is really what I want to find out: what should I put on the test?

Students can use internet, textbook, notes etc. All resources are fair game and ok.

stricklandg said...

I work in a one to one school and I like this idea. I was fortunate to have participated in a literacy grant that has taught me to try and write questions that assess understanding rather than knowledge.

An example of the difference between knowledge and understanding is a mechanic that knows all the parts of a car, but can't fix what's wrong with the engine.

If we write tests that assess what students understand about our content, there is no reason not to make factual information available.

To accomplish that goal I have had to move away from multiple choice assessments. I learn more about what students understand through open ended questions. They are a pain to grade, but such is the life of an educator.

Unknown said...

I teach in a school with essentially 4-day weeks, we get each class for about 40 minutes on a Friday twice a semester-so 4dwks. I usually don't want to waste time in class on tests, so they become brutal take-homes with some short quizzes in between. I've been accused of requiring more essay writing than the writing teachers. I like the opt out idea though. (less reading essays-what a job teaching LA) Really going to make a go of Flipping this year-not sure if traditional testing will makes it's way back or not-kind of guessing not. :)

Vlogger said...

First Started Packers and Movers in Kerala