Your first question regarding blue highlighted digits:
Option (a) can get cumbersome and can also give away the answer. It is best at this grade level to follow print, substituting italics for the blue highlighting--option (b). BUT NOTE THAT the termination indicator is not used to end the italics in a number. A new numeric indicator signals the return to regular type. See NC Sec.32.d. which is illustrated further in the lesson book (An Introduction to Braille Mathematics) in lesson 4, Sec.42.b.
Your second question regarding yellow highlighted vocabulary words throughout the book: It is important to retain distinction for these words. In the past we would have substituted the boldface type-form indicator for these words. In a literary transcription before BF2011 was published, the most common treatment was to use the boldface indicators. Since the literary boldface indicator is not used in a Nemeth transcription, we would use the Nemeth boldface type-form indicators in a math book. Although it is tempting to use the yellow highlight symbol, NC §35a specifically tells us NOT to use dots 456 with any other sign. Until BANA gives us a recommendation, we should continue to use the rules as we have done in the past which, in this case, is to use boldface type-form indicators for yellow highlighted words.
Please read the post nearby entitled"Colored Digits and Bulleted Lists" for Dorothy's answer to your question about color in Nemeth transcriptions.
Your final question has to do with general decisions made when formatting a textbook--you are smart to settle on a consistent treatment of these items when beginning a project. These items are headings. The colors are irrelevant to the understanding of the text. Keep it simple! In a math book, we get most of our heading rules from EBAE and BF. Here is one idea:
BF 4.8.4 suggests we consider changing paragraph headings to cell-5 or cell-7 headings. "A paragraph heading may provide better navigation for the braille reader when it is separated from the paragraph and changed to a cell-5 or cell-7 heading." I think this would work well with the "Mathematical PRACTICE 6" type headings. I have not been told that we cannot use cell-7 headings in a math book. Retain print capitalization as we do in any heading.
Then use regular literary italics for the boldfaced paragraph heading "Explain to a Friend."
BF 4.8.2 says "Italics may be used if all paragraph headings throughout the book have the same font attribute. This change is noted on the Transcriber’s Notes page."
For Question 17, you could devise an icon symbol for the red question mark, and pull it and the red paragraph heading "Building on the Essential Question" out as a cell-7 heading.
I invite other ideas or arguments.