We should really know the scientific name of the species of tree you are talking about to be sure about this. I recently learned about one called the Japanese Elm, Ulmas parviflorum . Is it this one? If yes, then it is safe a part of a varied diet.
Red maple (Acer rubrum) is very toxic to horses and there have been indications that others are affected also, but to much lesser degrees (no real concrete info that I can find..or should say have found with very limited research). Since many of the Japanese maples are red, I would personally err on the side of caution and keep them away.