Robotics 2
Final Challenge: Object Sorter
Final Challenge: Object Sorter
You have now made it through the entire Robotics 2 course, and you are about to test your complete set of knowledge by making a working robotic system. You've already learned everything you need to know to do this; you just need to put it all together. In this class, you get to pick one of three options for your Final Challenge. This option is called the 'Object Sorter', and it will mostly test your knowledge of machine vision and AI. You should start by setting up your camera looking down at your board. Here is what the final Object Sorter should do:
1. Place all of the color/shape pieces from the set in your kit on the desk, out of sight of the camera. Sort the pieces into two piles according to some rule involving shape, size, or color. For example, you might place all of the squares and all of the triangles in one pile, and all of the circles in another pile. Or, you might place all of the large objects in one pile, and all of the small objects in another pile. Or, you might place all of the red objects and all of the green objects in one pile, and all of the blue objects in the other pile.
2. Remove 2 pieces from each pile and set them aside. These pieces are the 'testing' set, and the rest of the piles are the 'training' set.
3. Run your Python code, then place all of the pieces of the training set for one pile (one class) on the board in view of the camera, then click a button or press a key.
4. Remove all of the pieces on the board and place all of the pieces for the other pile (the other class) instead. Click a button or press a key.
5. Remove all of the pieces on the board. Randomly select one of the pieces in the 'testing' set and place it on the board. Click a button or press a key, and the Python code should display which class (1 or 2) that piece belongs to. Record whether or not your algorithm was correct, then repeat for the other testing pieces. You should be able to get at least 3/4 correct.
In order to accomplish this, I recommend that you do the following five steps in order. As you go, TEST EACH STEP alone, then test each step working together with all of the previous steps. Don't move on to the next step until you have verified that the current step is working right.
There are five steps, and five in-class days to work on this. Try as best as you can to keep up with the schedule, and you should be ready to go by the time Finals Week arrives.
1. Place all of the color/shape pieces from the set in your kit on the desk, out of sight of the camera. Sort the pieces into two piles according to some rule involving shape, size, or color. For example, you might place all of the squares and all of the triangles in one pile, and all of the circles in another pile. Or, you might place all of the large objects in one pile, and all of the small objects in another pile. Or, you might place all of the red objects and all of the green objects in one pile, and all of the blue objects in the other pile.
2. Remove 2 pieces from each pile and set them aside. These pieces are the 'testing' set, and the rest of the piles are the 'training' set.
3. Run your Python code, then place all of the pieces of the training set for one pile (one class) on the board in view of the camera, then click a button or press a key.
4. Remove all of the pieces on the board and place all of the pieces for the other pile (the other class) instead. Click a button or press a key.
5. Remove all of the pieces on the board. Randomly select one of the pieces in the 'testing' set and place it on the board. Click a button or press a key, and the Python code should display which class (1 or 2) that piece belongs to. Record whether or not your algorithm was correct, then repeat for the other testing pieces. You should be able to get at least 3/4 correct.
In order to accomplish this, I recommend that you do the following five steps in order. As you go, TEST EACH STEP alone, then test each step working together with all of the previous steps. Don't move on to the next step until you have verified that the current step is working right.
There are five steps, and five in-class days to work on this. Try as best as you can to keep up with the schedule, and you should be ready to go by the time Finals Week arrives.
Once you have successfully found objects in view, you now have to get some features of the objects. We learned all about finding features here. This step is a little tricky, because you might not know right now which features will be useful for telling the difference between different categories of objects. At a minimum, you will probably need the average red, green, and blue values for each object and the total size of each object. For other features, you will have to get creative. You might want to try the height or width of each object. Keep in mind also that sometimes ratios of things can be more useful that the things themselves. For example, the height/width is known as the 'aspect ratio' of an object, which can often be a useful feature.
Another possibly useful feature is the amount of space that an object takes up within its 'bounding box'. A 'Bounding Box' is the smallest rectangle that can be drawn around an object. When you find the height and width of an object, you are finding its bounding box. If you multiply the height and width of an object and then subtract the object size, you will get the number of pixels within the bounding box that are not occupied by the object. This can sometimes be a very useful feature.
Another possibly useful feature is the amount of space that an object takes up within its 'bounding box'. A 'Bounding Box' is the smallest rectangle that can be drawn around an object. When you find the height and width of an object, you are finding its bounding box. If you multiply the height and width of an object and then subtract the object size, you will get the number of pixels within the bounding box that are not occupied by the object. This can sometimes be a very useful feature.
Once you have completed Step 2, you should be able to look at the data points representing the two classes of objects in 3D space (or 2D space, if you only have 2 features). Now, you can use an AI method to create a model of the difference between these two classes. We learned how to create models using two learning algorithms: Support Vector Machine and Artificial Neural Networks. You have to choose one of these and write it into your code. We learned how to do that here.
You might not know ahead of time which algorithm will work best for you, or what parameters will work best. Especially if you choose ANN, you will have to do some experimenting with the number of neurons in a layer and the number of hidden layers. This testing will be easier to do after you have finished the next step, so I've put this 'tuning' of parameters into its own step as Step 5.
You might not know ahead of time which algorithm will work best for you, or what parameters will work best. Especially if you choose ANN, you will have to do some experimenting with the number of neurons in a layer and the number of hidden layers. This testing will be easier to do after you have finished the next step, so I've put this 'tuning' of parameters into its own step as Step 5.
Once you have a model, you can use that model to predict which class an unknown point falls into. You will need to write some code to extract features from the 'testing' object, then put those features into the model to predict which class the testing data point belongs to. We learned how to do that in the same day we learned how to create the model, which is here. This code should output something indicating which class it thinks the testing point belongs to, such as '1' for the first class and '2' for the second class. You should also write the code so that you can test more than one object, one after another. Since you know which class the testing object actually belongs to, you can keep track of how accurate your model is. This will help you in the last step: tuning the parameters.
Now that you have the ability to create a model and test your model, you need to do some experiments. Anytime you are doing an experiment, you have two kinds of variables: independent variables and dependent variables. Independent variables are the things you can change, and dependent variables are the things you measure. In this case, your dependent variable is the accuracy of your model. For example, suppose you have 4 testing objects. Each testing object is either classified correctly or incorrectly by your model, and you should keep track of how often it is correct vs incorrect. Suppose that you did the test with the 4 testing objects 5 times (with a different set of 4 testing objects each time). If the testing object was classified correctly 15 times and incorrectly 5 times, the model has 15/20, or 75% accuracy.
Now, you will change one of your independent variables. Your independent variables include: the number of features you use, which features you use, the learning algorithm (SVM or ANN), and the parameters of the model (in ANN, the number of hidden layers and the number of neurons in each layer). Test the model with the new values of the independent variables to determine if the change is a good change or a bad change. Keep in mind that your system might work really well when you sort the objects in one way (for example, by color) but not when you sort in another way (for example, by size). But, you are trying to build a system that will work reasonably well (75% correct) for ANY way you might sort the pieces (by color, shape, or size).
Now, you will change one of your independent variables. Your independent variables include: the number of features you use, which features you use, the learning algorithm (SVM or ANN), and the parameters of the model (in ANN, the number of hidden layers and the number of neurons in each layer). Test the model with the new values of the independent variables to determine if the change is a good change or a bad change. Keep in mind that your system might work really well when you sort the objects in one way (for example, by color) but not when you sort in another way (for example, by size). But, you are trying to build a system that will work reasonably well (75% correct) for ANY way you might sort the pieces (by color, shape, or size).
Object identification is the part of the code that starts with an RGB image and ends with an Object Matrix. This step includes converting to grayscale, thresholding to convert to black and white, masking to remove noise, and use a connected-pixel algorithm to find all of the objects. We learned how to do all of this in the 'Masking and Image Segmentation' section.
To check if this is working correctly, try running your code and placing some number of objects in view of the camera. From the Python code, output both the black-and-white image after the masking and also the maximum number in the object matrix. The maximum number in the object matrix should match the number of objects you have placed in view of the camera.
To check if this is working correctly, try running your code and placing some number of objects in view of the camera. From the Python code, output both the black-and-white image after the masking and also the maximum number in the object matrix. The maximum number in the object matrix should match the number of objects you have placed in view of the camera.