Whenever I have to deal with a problem weed/plant at work I’ll do a google search on said plant and start clicking links from university or state agricultural extensions.
Round Up will work on smilax, but not at the concentrations you probably used. And like any plant with a large starch filled tuber it may take a few applications to act.
https://extension.uga.edu/publicatio...g%20Greenbrier