Underfitting, Overfitting or Bias and Variance

In Machine Learning, we often hear the problems of underfitting, overfitting, or bias and variance.
What are they and how to "diagnose" the problem of your models?

Underfitting / high bias

• Symptom: Your training error is high
• Problem: Your model is not able to capture the underlying structures/relationships in your training data
• Solution: Make your model more powerful (e.g., bigger nets, longer training time, more iterations)

Overfitting / high variance

• Symptom: Errors on your development set are much higher than your training errors
• Problem: Your model is "too fit" to the training data
• Solution: Use more training data (or data augmentation - e.g., flip/rotate images to have more training samples), add regularization to the model (or some techniques like drop-out, early stopping, etc.)