The standard advice is to always start from a block. A block is a tried-and-tested base pattern for a given fit category. It encodes the proportions and ease values you want to start from, so you are not calculating from scratch every time. Good advice, most of the time. But not always.
When the silhouette is too far from the block
If your design silhouette is dramatically different from the block — say, an extreme drop-shoulder with a very high armhole on a body block drafted with a standard set-in sleeve — you will spend more time slashing and spreading the block into the new shape than you would have spent drafting the new silhouette from measurements. When your design requires more than two major structural modifications, draft from measurements.
When the fabric is unusual
Blocks are calibrated for a specific fabric type and weight. A jersey block assumes the fabric will stretch and recover in specific ways. A woven block assumes no stretch. When your fabric sits outside the expected category — a very open, drapey knit; a heavy, stiff canvas; a bias-cut woven — the block's ease values are wrong from the start. You will fit and adjust and fit again, and eventually arrive at a pattern that looks nothing like the block you started from. Better to start from zero with the right ease calculations for your actual fabric.
When the garment is structural
Outerwear with internal structure — chest canvas, shoulder pads, wadding — behaves differently from the soft shell garments that most blocks assume. The internal layers add thickness and change the fit geometry at every seam. If you are not working in a category where structured blocks exist, drafting from scratch gives you cleaner geometry.
The block is a tool, not a rule. Know when it helps you and know when it is slowing you down. The most experienced pattern-makers switch between both approaches depending on what the design actually needs.