Vegetarian, high fiber, low fat diet detrimental for Testosterone

You guys know that in a month almost 98% of calories come from plant based dishes (only occasionally I might consume seafood or chicken and I have not consumed beef, pork, rabbit, etc. for many many years).  I also happen to consume an extremely low fat diet (while I have always been a healthy eater, I am also not a big fan of dishes cooked with a lot of fat).  And because I like oats, fruits, beans, and vegetables, I consume as much as 50 grams of dietary fiber daily (men should typically consume around 38 gm and men over 50 should eat about 30).

So what happens if you eat like me?  Well, when you eat a low fat diet and that too with fibers, particularly soluble fiber that comes from oatmeal, whatever little fat you ate is not absorbed by the body and is eliminated with the fiber.  Eating a lot of fiber is an excellent choice for overweight/obese men who need to lose weight but for skinny guys like me, fat is essential since it is the building block for Testosterone.  So for men like me oatmeal is not a blessing nor are vegetables and beans.  A study published in the Journal of Clinical Endocrinology & Metabolism found that a low fat, high fiber diet can lower Testosterone by as much as 12% in just two months.  While I have been trying to eat more fat, this week I have cut back on my oatmeal consumption by half and replaced it with one scrambled egg.  I am hoping that this will help me.  So for those of you who are vegans or vegetarians and limit your consumption of fat (and end up eating a fiber rich diet because otherwise there is nothing else to eat), this is a warning that you may have low-T and you need to very carefully watch how much fat you eat.  My hypothesis is that this was not a problem in my youth when the body was already producing tons of Testosterone but now that I am approaching 50, this has become an issue.