Machine Learning - Course handbook

This page is also available in frames.

Introduction

Lectures

Session times and locations

The first meeting will be the first lecture in week 1. There will be no second lecture and no lab class in week 1.

I'll be doing attendance sign-ups so please attend all lecture and lab sessions (or let me know if there's a problem).

Lab sessions will be used by students to work on the course assignments and to discuss issues from the lectures.

Course text

Good introductions to the area include the following.

• Principles of Machine Learning (Hand et al., 2001). Long winded and somewhat myth-ridden but also packed with information. Written from a mathematical/statistical viewpoint rather than a computational one.

• Predictive Machine Learning (Weiss and Indurkhya, 1998).

• Introduction to Machine Learning (Tan, Steinbech and Kumar, 2006). Probably the most up-to-date and comprehensive general reference. We have a copy in the Informatics library.

• Machine Learning: Introductory and Advanced Topics (Dunham, 2003).

• http://www.thearling.com/text/dmtechniques/dmtechniques.htm Nice, brief overview of DM techniques and issues. There is also a complete tutorial from the same guy at http://www.thearling.com/dmintro/dmintro_2.htm

• http://www-users.cs.umn.edu/~kumar/dmbook

Programming assignment

In 1984, an animal behaviour researcher studying the social behaviour of primates was killed in a car accident while returning from a field trip. In his belongings, police discovered several listings of data but the notebooks, assumed to contain details about the origins of the data, were all burned. The longest of the listings is reproduced at ex1-data.txt. This is in tabulated form and it is generally believed that each line records the values of several variables, followed by a value which is either 1 or 0. This is believed to be a classification value of some sort but the nature of the class has never been determined. The final 100 lines in the listing are missing this classification value.

The exercise involves (1) analysing the properties of the data, (2) implementing (using Java or similar language, not matlab!) and then using one or more data-mining methods to derive predicted classification values for the final 100 cases and (3) submitting predictions by email and (4) presenting the results of the experiment in a report.

Marks for the assessment will be awarded using the following scheme.

• 5% for a clearly written introduction to the topic of general field of data mining. (Approx half a page.)

• 5% for a clearly written introduction to the method(s) used in the experiment. (Approx half a page.)

• 10% for informative analysis of the training data. This could be based on either formal methods (such as analysis of correlation, density, variance etc) or informal methods (such as reasoned deduction about the likely nature of primate research). Ideally it should be based on elements of both. The aim should be to show what assumptions can be made about the data and explain how those assumptions might inform the choice or application of machine learning procedures. (Approx 2 pages.)

• 10% for clear presentation of results. This section should show what output your program produced and how it is to be interpreted. It may make use of tables and graphs to show how well the program performs. (Approx 3 pages.)

• 10% for level of functionality in the program.

• 10% for program design.

• 10% for concluding section and appendices. The conclusion should discuss any limitations of your approach. There should be a self-assessment (in terms of the given credit factors) and a full bibliography. You must directly acknowledge any quoted material. The code and other relevant data (e.g., graphs) should be included as appendices.

• 30% for the accuracy of the predicted classification values. Marks for this component will be normalised appropriately. The most accurate predictions in the group will score the full 30%. Other marks will be scaled accordingly. But if the prediction error rate is worse than 50%, the mark will be zero.

• 10% for material that goes beyond the spec, e.g., evaluation of performance of a second algorithm.

The full report should be handed in as indicated. However, The predicted output values should be sent seperately to me (C.Thornton@sussex.ac.uk) as a text attachment (i.e., a `.txt' file) sometime before the submission deadline for the report. The email message should have the subject label `DM exercise: Joe Bloggs', except using your first name and last name. The attachment should be a listing in which each line contains the predicted classification value for one of the 100 test cases.

The output values must be in the same top-to-bottom order as that used in the original data file. There should be no blank lines and no tabs in the file. In the case of multiple submissions, the final submission will be taken as definitive.

Please take careful note of the requirements. In particular

• DO send the data as an attachment.

• DO make sure the attachment is just an ordinary .txt file (i.e., an ascii file).

• DO NOT include the input values. There should be just one value per line and this should be a 1 or a 0.

• DO NOT put commas or any other punctuation in.

• DO NOT insert any blank lines.

General guidelines for the writing of program reports are below. However, the section requirements above should be treated as definitive.

Usual penalties apply for late submissions.

Students resitting the course should make sure they implement a different method this time around.

Guidelines for producing the report

1. INTRODUCTION

This should be a general introduction to the theory, techniques and general area of work that the program relates to.

2. ANALYSIS OF THE TRAINING DATA

There is no `one right answer' for this section. Your analysis could be fairly formal, involving statistical analyses of correlation etc., possibly making use of Excel scattergrams etc. Or it could be more informal and rely on visual inspection of the data and the making of intelligent assumptions about the task from which the data were obtained.

3. RESULTS

This is where you present the results of your study, e.g., levels of training and cross-validation error achieved by your method(s). You may also want to show some sample predicted outputs. You may want to talk about how your program(s) deal with the anomalous cases in the data. If you've used the kNN method and employed cross-validation to determine the best value of k, this is where you would present a graph relating values of k to cross-validation error.

4. ILLUSTRATION - Illustrations of your program(s) working. This could be based on an edited output listing, with annotations to show what is going on. If the program uses a GUI, it could be based on a sequence of annotated screenshots. (If necessary `screenshots' can be drawn by hand.) However, Edit out repetitive chunks from any program output and edit in explanatory comments. Annotate the most significant bits of the output to draw attention to them.

5. SYSTEM OVERVIEW - Explain precisely HOW the program works.

You should discuss the main steps the program goes through to achieve its results. You do not need to describe every line of the program. You should give an overview of what sorts of things are represented, how they are represented, where input comes from, how it is transformed, where output goes to, etc. The overview should refer to a copy of the program attached as appendix-1. (See below) In particular make sure you describe what kinds of objects are represented, and how you represent them (e.g. using lists, or vectors, or numerical values for variables or whatever). Explain what problems your program had to solve, and how it solved them. Use figures and diagrams productively so as to back up the text. Do not rely on an automatic documentation facility (e.g., javadoc) to write the system overview for you.

When writing the overview, don't write as if you are communicating with a tutor. Equally don't write as if for a novice. Try to write for an audience consisting of fellow students who may be a week or two behind you in their understanding. When describing what a method does, always adhere to the principles of modularity, i.e., start by stating what sorts of inputs the method takes, and what sorts of output it produces. Give an explanation of the relation between input and output, and one or two simple examples to illustrate. If there are no inputs, or no outputs, then say so.

5. CONCLUSIONS

This is where you should assess what you have achieved and discuss any limitations or problems. It's also the place to talk about potential future developments. Don't be afraid to criticise your own program. If you have read about similar programs, or related programs, it may be appropriate to include some comparisons. What lessons, if any have been learnt? Were your goals achieved? Is there anything you now think you should have done differently?

6. SELF-ASSESSMENT

This is where you present a formal evaluation of the whole submission in terms of the specified marking criteria. For each criterion, note how well you think you've done (and why) and give yourself a mark.

7. BIBLIOGRAPHY

List books, articles, web pages, files etc. considered to be relevant.

8. CODE APPENDIX - The whole program, with comments explaining what the classes represent, what the methods do, what the global variables (if any) are for, etc.

References

Hand, D., Mannila, H. and Smyth, P. (2001). PRINCIPLES OF DATA MINING. Cambridge, Mass.: MIT Press.

Tan, P., Steinbech, M. and Kumar, V. (2006). INTRODUCTION TO DATA MINING. Boston: Pearson/Addison Wesley.

Weiss, S. and Indurkhya, N. (1998). PREDICTIVE DATA MINING: A PRACTICAL GUIDE. San Francisco: Morgan Kaufmann Publishers Inc.