Spousal Privilege covers confidential communication between a husband and wife (including oral, written and/or expressive action). It typically does not apply to situations where a spouse or child was a victim of a crime by the other spouse, but only as to that crime. It also doesn't apply if third person were present (irrespective of whether their presence is known or unknown). The privilege is available to the spouse against whom the testimony is offered (meaning your husband's attorney will have to raise the issue), although most courts feel that consent from both spouses is necessary for a valid waiver.
The questions are did a crime occurr against you or your children (sounds like the prosecutor thinks there was), whether the court rules that an allegation is enough to compel you to testify, was there a third party present, is there prima facie evidence of a crime against you and/or your children.
Has your husband retained an attorney?