Thank you for your question.
First, you’re correct that Nemeth Code says to format instructions in 5-3. A second paragraph in instructions should be formatted in 5-3. I suppose you could choose to condense all of your instructions to one paragraph if you can do so consistently throughout the book.
There is no hard and fast rule for how to represent the colored rectangles in your example (thank you for the attachment). Perhaps you could use other shapes to represent the other colors of rectangles. If you do that, you must explain it in a key for the reader.
Please let me know if you have follow-up questions or concerns.
–Kyle