Forward swept wing?
Now that I think about it, the tail booms may have a beneficial overall effect for your design. The deal with forward swept wings is that when flying, they will want to twist up and/or down and need to be very torsionally strong so as not to diverge. However, your tail booms mounted were they are will counteract the torsional forces on the swept forward wing. Therefore, you might not have to build your wings any more torsionally resistant (this will save weight) than a straight wing if the booms are not too long as to overcompensate. You may have to lengthen or shorten your booms (or use less hstab area) for optimum torsional stability though. How long is that? How much hsatb area? I'm not sure how to figure that out. All I can think of are trial and error methods, but I'm sure there must be a way to compute this.
The weight of the retracts will slow the roll accelleration but that will not make the design unflyable. It will only affect roll acceleration (and deceleration) and will not affect maximum roll rate itself. I think you have a good design here worthy of exploration.