Difference between revisions of "Diamonds Regression Lab 202"

Latest revision as of 22:10, 31 January 2020

Regression Analysis

The size of a diamond can described by its dimensions (x, y, and z) and by its weight (usually expressed in carats). We are going to investigate the relationship between these two descriptions, from a data set describing thousands of diamonds. Jewelers often measure these characteristics of diamonds when they set the diamond's price, but jewelers don't always make the measurements accurately. An understanding of the relationship between these variables, and what is atypical, might alert a jeweler or a buyer to the possibility that the measurements were taken incorrectly.

The weight of a diamond is its density times its volume. As you might imagine, the shape of a diamond matters. If diamonds were cut as cylinders, the relationship between weight and dimensions would be given by the geometry of the cut:

weight = density * pi/4 * x*y*z.

If diamonds were cut as right-circular-cones this relationship would be given by a different geometry:

weight = density * pi/12 * x*y*z.

In both cases, the weight is the density times a coefficient (a number) times the product of the dimensions. In both of these cases, the diamonds would be pretty ugly, but the point is the coefficient would be the same for diamonds of the same shape, and different for diamonds of different shapes. You can assume density is the same for all diamonds.

You are going to find this coefficient for the diamonds of a data set linked below, which presumably have a similar shape (the data code book says they are all "round cut"). Note the different diamonds in the data set may be cut slightly differently, with imperfections or peculiarities, so you should expect scatter in the data.

You want to find a formula that predicts weight in terms of x, y, and z (listed in a data set). Your formula would be very useful for a jeweler!

Step 0: Data Acquisition

Download the diamonds3K data. Once you have saved the file, you can load the data into StatCrunch. Note that it is not recommended to load the data into Excel first because then you lose the column headers. You should also view the codebook a larger data set, from which our data were randomly sampled (so they would fit reliably into StatCrunch).

Step 1: Feature Engineering

We have three predictor variables x, y, and z. But for simple linear regression, we can have only one explanatory variable. Therefore, we want to create a new feature that combines the information from all these variables. As suggested above, a good feature might be

x*y*z

For comparison, we are also going to use the following feature in a separate model:

x+y+z

Go ahead and create these columns in your data set. Use Data --> Compute --> Expression in StatCrunch.

Step 2: Regression

Use simple linear regression (Stat --> Regression --> Simple Linear) to compute b0 and b1 for the following linear models:

PRODUCT MODEL: Carat = b0 + b1 "x*y*z"

SUM MODEL: Carat = b0 + b1 "x+y+z"

Here "x*y*z" and "x+y+z" should be whatever you called the columns in the Feature Engineering Step. If you didn't specify labels, they will be labeled as shown here, by default. Record the slopes and intercepts for later reference. Keep a record of all your results to write up later.

Step 3: Model Selection

Make a judgement concerning which of the two models is more appropriate for simple linear regression. Remember the conditions listed in the book, and how you check for them. Don't worry about outliers yet, we will deal with them, below. Be prepared to justify your judgement when you write up your lab. Be sure to check the residuals-versus-X-values plot as well as the scatter plot. Also compare R^2 for the two models. Again, keep a record of your results to write up later.

Step 4: Data Cleaning: Outlier Elimination

You might have noticed outliers and remembered that one of the conditions for regression was the "no outlier condition." You can eliminate outliers, but only if there is a justifiable reason to do so. Consider an outlier. If the jeweler measured the dimensions wrong, and there is no way to tell what the dimensions should be, then yes you should eliminate the diamond from consideration. On the other hand, if the store actually sold a round cut diamond with said dimensions and said weight, you should not eliminate the diamond from consideration.

How can you tell if the dimensions are wrong? In StatCrunch, click on the most extreme outlier. It will turn pink on the scatter plot and other graphs created. Now plot three scatter plots showing (x, carat), (y, carat), (z, carat) and locate the pink outlier on the graphs. Could you change a single measurement of the four (i.e. x, y, z and carat) without affecting at least one of the others? What does this tell you about the validity measurements? It would be too tedious to check all of the other outliers (but if you were working for a client, that's what you would have to do) so we are going to assume that all other outliers tell the same story. Looks like the jeweler could have used your formula!

Now we want to eliminate all outliers at once. Redo the regression of the chosen model and save residuals (that's an option in the Regression dialog box). Now redo the regression again and specify "Where abs(Residuals) < 'Threshold'". Here 'Threshold' should be some number that you get to choose, but choose appropriately, to eliminate outliers with large (positive or negative) residuals that stand apart from the main part of the graph. To help you decide where to put the threshold, look at three plots: First, look at the residual versus x*y*z plot (here x*y*z is what the StatCrunch calls "x"). Now, each of these residuals has a value which you can save in a column, putting each residual in the row corresponding to the data point of said residual. In the Regression dialog, look under "Save" and choose "Residuals" then press "Compute". Once you have the column of residuals, you can look at the second plot: the histogram of the residuals, and the third plot: the box plot of the residuals. The is no one correct Threshold, it is correct as long as you can justify it.

Now compare the R^2 value with and without outliers. Compare your slopes and intercepts with and without outliers. How different are they?

Step 5: Model interpretation: Intercept

What is the intercept of the final model (after outliers are removed)? What is the meaning of the intercept? Is your intercept zero? Should it be, or rather how much would a round cut diamond weigh if the product of its dimensions was zero (i.e. one of its dimensions was 0)? What does the model predict for the weight of the diamond if x*y*z=0?

Would more data from the same distribution tell the same story?

Just read this subsection.

The first part of our textbook contains little burbs called "Random Matters," where the authors address the question, in various ways, is an apparent result we see due to randomness or something systematic? In our context, the question would be: if we observe more diamonds from the same distribution would they tell a consistent story concerning the intercept. True, because there is random scatter in the data we would not expect the intercept to be exactly zero. But would the sign of the intercept be more-or-less consistent? StatCrunch gives you a number which you will be able to interpret later on in the course: the answer is a resounding yes: more diamonds would tell the same story---the intercept would continue to be non-zero in the same direction. Ask your instructor for details if you are curious.

Do assumptions of regression affect the story told by the intercept?

Consider the residual plot (Residual versus x*y*z) after outliers are removed. What does that plot tell you about the model? Does this plot satisfy the "Does the plot thicken?" condition? Could this failed condition tell us that we cannot interpret any conclusion drawn from regression (as explained above) to say that the zero-volume diamonds have non-zero weight?

How can we fix our model so that the intercept is as we expect?

From now on, when you write the model, drop the intercept from the model. Optional: In StatCrunch there is a way of recalculating the slope under the assumption that the intercept is zero. It involves choosing "Polynomial" Regression, choosing "Polynomial Order = 1" and checking "Without Intercept". The answer you get for the regression slope is almost exactly the same, with or without the intercept, so it's OK to just write the slope term you get with Simple Linear Regression, and don't write down the b0 intercept term. Feel free to get a possibly more accurate slope, though.

Step 6: Model interpretation: Slope

What is the slope of your model, b1?

Plug in your answer for b1 into the equation below and solve for "a".

Weight = b1*x*y*z = Density * (pi/a) *x*y*z

Density = 0.01755 carats/mm^3
pi = 3.1415

Now compare your answer for a with the values for cylinders (a=4) and right-circular-cones (a=12), shown at the top of the lab above.

Step 7: Your tool in production

Going back to the problem of when to flag a given diamond as possibly measured incorrectly, we are going to use your model; plug in b1:

Carat = b1*x*y*z

We are going to measure x, y, z, and carat for the diamond, then compute a residual. If the residual is too high or too low we are going to flag the diamond. You choose what residuals are too high and too low based on your understanding of the data. It may be reasonable to choose the same threshold as chosen in step 4. Based on this choice, write a formula indicating which diamonds are flagged involving x, y, z, b1, and carat, and inequalities.

Step 8: Testing your tool

Compute the residual in Step 7, for each diamond in the data set. Use Data ---> Compute Expression.

Count the number of diamonds in the data set that would be flagged as possibly measured wrong. Use Stat ---> Summary Stats. Choose n (count) as the statistic to be computed, and set a where function based on the residual above to include only those flagged. What percentage of diamonds are possibly measured wrong?

Step 9: Publication

Turn in your report that explains what you have done. Include a short description, with headings for each step above.

Please include in your report:

• Show the scatter plot of carat versus x*y*z for the whole data set.
• Show the scatter plot of carat versus x+y+z for the whole data set.
• Give the justification for your selection of model in step 3.
• Show the scatter plot of carat versus x*y*z with the outliers removed. Justify your choice of threshold for removing outliers.
• Values for b0, b1 and R^2 for model Carat = b0 + b1 * x*y*z with outliers removed.
• Show Residual versus x*y*z plot with outliers removed.
• Report final model dropping b0. Report your value of "a" in step 6.
• Report chosen threshold for flagging diamond as improperly measured (all without b0).
• Give the inequalities for diamonds that are flagged and not flagged as improperly measured.
• Show an image from StatCrunch of count in data set of flagged diamonds.