Skip to main content

Underfitting, Overfitting or Bias and Variance

In Machine Learning, we often hear the problems of underfitting, overfitting, or bias and variance.
What are they and how to "diagnose" the problem of your models?

Underfitting / high bias

  • Symptom: Your training error is high
  • Problem: Your model is not able to capture the underlying structures/relationships in your training data
  • Solution: Make your model more powerful (e.g., bigger nets, longer training time, more iterations)

Overfitting / high variance

  • Symptom: Errors on your development set are much higher than your training errors
  • Problem: Your model is "too fit" to the training data
  • Solution: Use more training data (or data augmentation - e.g., flip/rotate images to have more training samples), add regularization to the model (or some techniques like drop-out, early stopping, etc.)
Labels:

Comments

Post a Comment

Popular posts from this blog

SAXParser: too many exceptions for invalid XML character..

I'm working on my Similarity Search project, in which I have to implement the Tree Edit Distance and Traversal String Edit Distance. Trees are all represented in XML format and I'm using SAXParser to parse those XML files in java. I've used it a lot of times before but still, I don't quite like. So my first step is to create a valid XML database. However, "valid" to be parsed using SAXParser is complicated!! Here is what I get again and again: File Read Error: org.xml.sax.SAXParseException : The content of elements must consist of well-formed character data or markup. org.xml.sax.SAXParseException: The content of elements must consist of well-formed character data or markup. The reasons can be different, like: - Tags cannot contain number (e.g., is an invalid tag) - Tags cannot contain some symbols, like {, ., ?, etc. ("_" or "-" is fine) - Tags cannot be empty However, in my database, all of the tag are numbers.. To make it a valid XML fi...
2 comments
Read more

Sigmoid, tanh, ReLU functions. What are they and when to use which?

If you are working on Deep Learning or Machine Learning in general, you have heard of these three functions quite frequently. We know that they can all be used as activation functions in neural networks. But what are these functions and why do people use for example ReLU in this part, sigmoid in another part and so on? Here is a friendly introduction to these functions and a brief explanation of when to use which. Sigmoid function Output from 0 to 1 Exponential computation (hence, slow) Is usually used for binary classification (when output is 0 or 1) Almost never used (e.g., tanh is a better option) Tanh function A rescaled logistic sigmoid function (center at 0) Exponential computation Works better than sigmoid ReLU function (Rectified Linear Unit) and its variants Faster to compute Often used as default for activation function in hidden layers ReLU is a simple model which gives 0 value to all W*x + b < 0. The importance is that it introduces t...
Post a Comment
Read more