so our last next speaker is Dr uh Yong t uh Dr T is a um research scientist director at meta J he's also leading uh Lama uh reasoning group uh team
and um and I read quite a lot of papers from uh Dr T and early I think he's he had a project um this uh open uh goal project which is uh um sort of uh
you know looking at R for zero I think um Dr also working on uh multi-agent learning I think uh the
recent paper about large language model um recently I think it was about post tokens so the idea of say maybe we can do this uh reasoning not
necessarily in the language domain but maybe in the latent domain I think that's definitely a very interesting uh direct direction to look at it um so
please welcome uh Dr 10 um to tell us bit about this uh how can we unify you know the neuro at the symbolic
uh decision making
please uh can you hear me yes okay cool yeah thanks for introduction uh Dr Wong uh yeah so uh my I'm a resarch scientist director in manag gen right
now uh I have been in fair for 10 years and recently uh We join uh the manag Gen team I'm one of the research lead but not all of them so I saw there
actually many uh some resar lead as well on the reasoning s for Lama uh so I just joined for three
weeks so of course there's lots of things I don't know to some exent so I don't treat me as a for flagged uh lead for that uh for that for that
efforts uh so right now uh so we are busy with all the all then release right now uh hopefully things are fine uh so uh today I'm going to give a talk
which is mostly academic so towards unified framework of neuron symbolic decision making uh so uh
basically like the direction here that we are want to work on langage models and we all know langage model is very important and there's a lot of
applications so in this uh in our group we mostly focus on reasoning and planning right so um and when we work on reasoning planning we often ask
ourself the questions like what kind of problems that lar model cannot do very well the answer is of course
like the planning problems right so for the Trav planning problem that I often use as one example um The Lang model actually cannot do very well so
Tri plan is very simple you ask the user to provide a request uh to the language models and so that the language model can give you the travel
itinerary uh that s all this request right so for this kind of planning problem um usually people will think
about this kind of gentic behaviors right the will first collect informations and then uh give the plan according to all the uh all the constraint
budget constraint as well as the uh as a kind of Common Sense constraint and put them together uh into a plan right so that's actually the idea right
so uh of course like uh if you ask the model to do that um it's actually not doing very well um this is
actually one of papers that we published last year uh early uh last year it's kind of an old paper according to the uh the according to L model kind
of speeding U Pace right so I mean one year is like one century in the community of lar models so what happen is that even the soda models struggle
for hard planning problems right so uh at that time the best model is the gbd4 turbo and you what you
actually see is that if you ask the model to do planning for the travel um you actually only give you like 0.6% accuracy for the final pass rate which
means that the final trajectory uh is completely correct and satisfy all the constraints of the humans and also of the Common Sense constraint so it's
actually very low so if you even use um you allow the model to access to the ground Tru tools and
you have all the background informations and you don't need models to actually exract them you ask only ask the model plan then the model only give
you uh 4.4% of final pass rate which is actually uh also very low right so I of course like people will say about R1 talk about o right so currently
we only have o1 preview numbers uh because this is a out of day slide to some extent uh but with open
preview you only see the performance basically go up a little bit but not too much right so you only see the final pass to be 15% which is not ideal
uh of course it's much better than the 0.6% but it's still not super high compared to um if you think about like if the problem can be solved by
symbolic solvers same Trend happens to other kind of large models as well right so if you check the
previous Gemini 1.5 uh reports and what you see is that they are trying to test our their approach on the planning and you see when the scale of the
problem becomes higher uh larger and you see the performance all drop to zero right so this also happened to this plan bench uh which is from the ASU
and they also report that um the OM preview approach is not doing well when the plan become more
complicated right so recently we also we also have another paper called GSM infinite and that paper actually also show that the same thing also
happens to all the St models including DS models right so you see this um this kind of Decay when the problem become complicated so it's not like uh
the model is solving this kind of planning problem with test time compute but uh you still see the problem
there okay then the question is really is like what is the solutions R our Solutions right so there are several different way of doing that first of
all of course you can put more and more data into the scaling laws and uh and hopefully the scin law will help and the second is that we want to
leverate like test time scaling right so we can uh use l model plus tools L model plus CH of ss and
hopefully with this test time scaling test time uh usage of uh different uh vehicles and we actually end up with a solution uh that um uh that can do
better than lar mod alone and finally I will briefly introduce our recent uh theoretical studies I try to unify the symbolic structure from the uh
with the neuron network uh representation so you can actually find there actually symbolic structures
emerged from the training of neur networks using green descent and by doing that you actually see unification of the symbolic representation and neur
representation in the same framework but this is a very rudimentary and U and hopefully this might be in the future become mainstream later but we
don't know so I'm going to cover all this aspects so of course like in option one you need a lot of
compute a lot of data and it's very expensive right so um we are not sure where it works on reunion planning and option two is the test time scaling
so here I'm going to talk a little bit about these two parts first is to use and second is using channel of SCE so for Tool use uh what we do is that
um we actually like can call external Tools in order to uh address the problem of reasoning right so
uh for the we just take this uh traval planner problem as one example right instead of having the user to directly uh to having the model directly
answer the question provided by user what we can do we can detect the tool the tool basically say that um you start from the user request and you
first use l model to translate that into a symbolic form so this symbolic form basically characterize
accurately what the user really look like and together with the flight of hotel information um from like external world you can basically put them
together into a big like optimization problem and you solve that with the mixer Interlinear programming which is a citoral uh solware and that solware
will give you the symbolic representation of the optimal itinerary and you can basically translate uh
that back using the existing large models and then you get answer provided by the agent right so by doing that you we are able to give you a
guaranteed solution and at the same time it's not that slow it's actually pretty fast uh if you go through the entire pipeline so we actually build a
demo on top of that we actually show that with two or three seconds you're able to get a solution that's
guaranteed to be correct uh given the users's request I mean also given the condition that the translation uh with the larg models is actually pretty
accurate so you can see there's a two or three seconds um delay but the overall demo can still be run and we also test with our human annotators and
look like they are very positive about the results of course like if you think about the tri planning
uh we also want to ask let the agent to clarify uh questions right so we want agent to really think about like what are the details that is missing
from the user request and the user when whe whether the agent can actually ask more and more question to clarify that to make sure that they
understand so this actually motivated us to work on followup paper on like how to uh build this multi round
dialogue uh so that agent can uh interact with humans uh more effectively so we basically Build That by asking agent to follow the agent Constitution
right so basically the agent need to be evaluated according to several criterias and some criterias are actually very objective some Pro is very
subjective for example the credibilities so objective means that the agent need to finally get the answer
that answer is accurate and also uh the agent need to be very proactive in asking the right questions for that particular uh person of that person
right so agent also need to be efficient in answering asking all the questions uh before it get into the actual Solutions yeah so we basically do this
intutive DPO and U uh and reject sampling uh in two2 Manner and we' show that the a Ab models and
Trend this manner can be uh much better uh compared to the original 70b model that does not go through this training in term of multiple access okay
so this is the first part that we actually can leverage a tool to solve lar the complicated planning problems and the second part is that I going to
talk a little bit about lar model plus CH of thoughts and how to construct CH of thoughts to solve
this hard planning provin so uh in order to uh start with that let's start with this first paper that's called a search forer so in this paper we
construction of thoughts uh using the trace of existing citoral solvers right so when we think about this maze navigation problems right we have
starting points and then we have ending points and we ask the model to find what is the shortest path between
the starting and ending point we can we can either ask human to annotate or we can ask the symbolic solvers to uh to formulate the problem and uh to
make the problem uh to solve the problem while doing a kind of symbolic search so what the a problem what AAR will do is that it will open a priority
table the table will basically like find what is the uh right uh next uh sales next states of the
search to be explored and then following the way to for exploration then we finally find a path of to the destinations right have guarantees um and
then that it will guaranteed to give you the optimal trajectories so following the a star trajectory uh the search process we can actually construct
the uh the the construct the channel of ss as follows right so we start with the prompt which is the
representation of the problem specification then we can uh construct the trace by basically like a dump all the AAR search procedure so every time the
AAR search create a new node and in the search space we actually put a crate uh in the trace right every time a star actually close a node then we put
a close there so we basically put this long tra there and then finally once a star start to uh
find a goal we basically put a plan which is outputting the actual final trajectory which is the optimal trajectory so this basically uh give you this
basic uh framework of the search argumented model right so uh we have a prompt as the input and we also have asked the model to generate the trace as
well as the plan this is in contrast with the solution only models uh in which we start with a
prompt and then we ask more directly output the plan right so and of course the number of tokens will be very different right so this paper was
basically published uh early previous years uh 2024 in in that in that year people haven't realized too much uh the powerful of using very long chain
of thoughts uh to solve complicated planning problems but now I think everyone actually realize and people
are embracing a very long traces uh for solving the problems so uh we find that using this uh Trace augmented search a models what we happen is that I
mean the scaling law is very different right so we actually see that if you only use solution only models then you need a 1 million samples in order
to achieve Almost 100% accuracy but um if you use sear ARG models and use uh you actually use 10x
fewer data and at the same time you use 10x fewer parameters in order to achieve similar performance so the search augmented model is much more sample
efficient and also much more parameter efficient so this happen to the be the same for other kind of col plan task like soang in which like the agent
need to push the boxes to the destinations but it requires very careful planning because if the box
was being cornered uh there's no way back and you have to basically back Trace okay so uh once we have the orent models which is basically done by
imitation earnings one thing we can do better is by doing fine tuning on top of this uh this SE augmented models so what happened is that um we can f
tune the AUG the models so that it will give you shorter traces but still leads to Optimum plan so this
is a very typical reinforcement task because the goal is clear but how we achieve the goal we don't know right so they we ask the reinforcement agent
to basically like search around and find the solutions yeah that's the search form and by doing search form we actually see that it actually end up
with like a better model that has a shorter search Trace compared to original AAR search because of
this inuitive ading which is a PO man version reinforcement learning so uh we actually see this happens and uh in term of the numbers we see that the
ratio between the reduced trace and original uh a star Trace is getting higher and higher right so which means that uh this is indeed the case uh
numerically not only that but we also find that the not only the tra become shorter but also uh the
performance after fine tuning is also getting higher right so the ratio of the solution being up Optimum is getting higher and higher after each of
the iterations so this basically and after three iterations you already see that the model with 45 parameters Min parameters is actually comparable
with the model Trend with uh with like a 700 parameters and Trend with like a much larger data set so
this is actually very interesting and uh I believe that this is one of the very few papers at the beginning to show there's actually scen law uh on
test time and also on the training on the uh on the like a reasoning data that can substantially improve uh the performance of the reading tasks how
about the generalization I be very interesting to say you know the language model could generalize that
to some other domain which haven't been trmm now yeah so I think we have a tried several generalizations right so for this paper there's no much
generalization going on because we only train on the soan or on the on the maze uh but in the followup papers we actually see that things can generate
a little bit yeah for example uh in we can generalize to like different siid of the Maes that's that's
actually one way of thinking about that and we put like a multiple uh and we can also like try this ideas to for example Mass problems and it can also
generalize and give you better Solutions on mass problems okay good so for the trace if you sort of maybe generalize that the trace a transform that
tra into sort of a have certain semantic meaning like I don't know where some kind of language yeah
would that would that help in terms of generalize to maybe math problem or because all this search principle they are essentially the same right you
can learn uh I'm actually not sure because uh what we actually do is like in other side we want to start from like Tex based tokens and then we can
basically make it a latent tokens or lat canuse representation and then this might be more efficient h
yeah it could be you could down search in the in the latent space um yeah yeah uh sorry have you guys I mean I mean we're working on embodied AI a lot
right so have you guys tried to develop a VM version of this like something image of the environment because it might be nice for planning in embodied
environments actually yeah I think we haven't done that yet uh but I think that should be quite
straightforward right so yeah you just follow the same path but you basically render every stat as an image right so in that case like things might be
a little bit easier so for example give cot as like images instead of so here you have traces right so this actually not that intuitive right so if
you can render uh images uh you here is like the current status of the search uh space and then you
basically stack this images together as the visual channel of thoughts um this might also work well yeah and but then we have a long context issue
right because yeah so of course like you can always do some kind of compression on the yeah I'm going to talk a little bit about like how can we use
vqv to do that um okay thank you yeah so okay so that's the search for paper and uh once we have search
for paper then we have follow-ups right so we have one paper called dual forer uh in that paper uh we basically try to reduce the size the length of
the the tra being used for training right so you can randomly like dropping tokens uh for the search traces and this dropping tokens you can have like
a multiple way of dropping tokens you can drop like for example all the closes all the caes with all
the cost right so you can also even drop all the entire trace this means that we essentially train on mixture of the solution only and the search
argumented data together and this training actually turned out to be very interesting and actually give you some model that is um uh very uh is
powerful right so we actually end with this de forer uh model that I can automatically switch between the fast
mode which means that you directly give the answer and the slow mode uh which is basically give you chain of ss and then the answer right so you can
you can find you can directly you can have a model that can do both at the same time and at the same time it actually work better for dedicated models
on either modes so this is actually very interesting the way we switch between the two is very
simple right so you just um uh you just ask the first generated token of the Dual form to be either a beginning of search token or a plan token that
will end up with a direct uh optimal answers right so you only basically fix the first tokens and the model will behave very differently and uh it
would behave either like a faster mode or the slow mode forance right so it's it's actually interesting
to see that it's actually better than fast mode alone and better than small more alone so that's that's an interesting findings after we train these
models and we actually noticed that deep seek also show like similar behaviors uh there are some tricks in the Deep SE sying mode right so you can
actually uh make sure that if you have two space after the syncing tokens uh then the model will tend to
be a little bit longer in term of the syncing sequences if you have if you have not two sped but two uh two returns if you only have one returns after
syn after the syncing tokens then the model will tend to be a little bit longer in term of thinking process or vice versa I didn't remember which one
which but you can definitely check uh the interesting structure there I think this probably because
of the data they use to train a model will have this kind of specific structures uh then this will actually be learned by the models in order to
control the model um Model Behavior as the side effects and we actually happy to see that the Sim Behavior happens in this St out large models yeah we
also apply the uh deep the Dual form into like Mass problems right so the Baseline will have a very long
ter of thoughts but once you randomly droping like sentences and then use this to do fine tuning we surpris very surprised to see that the length of
the uh sequence will have a very basically the syncing tokens are very short become much shorter in the same time it's still reasonable right so and
it will give you the answer that's correct it's kind of interesting behaviors we actually do this Mass
tuning uh on the base models these are very they base models they're not mass specific model so the performance is not great but you still see that
the performance goes up a little bit right so after uh you do this uh do this dual formal training right so performance actually go like a little bit
yeah okay so uh this are basically conclude like our work on the token space and then we're going to
talk a little bit about our recent work that try to get into like this latent space so one of the paper that uh actually went viral last year is the
coconut paper in which like we're trying to replace uh the discrete tokens with the continuous tokens so that we actually and arrive at like a shorter
uh chain of thoughts and at the same time the chain of the The Continuous version of chain of
thoughts may have interesting properties that we don't expect and we see it might be interesting to explore further right so the idea here is very
simple right so traditional Channel thoughts has discrete tokens as intermediate stages of thinking but the why here we want to try to basally
construct like a continuous tokens to do this thinking process uh very simple so the training procedure is
actually uh straightforward right so we start from this state zero which every step is a uh every step represent a sentence uh of the three tokens to
do reasoning towards the answers so every time we do the training we do curriculum training we start from stage zero and then we gradually replace
every step into uh with the actual sort vectors right so we do that and uh we turn on stage one for a
while and until it until like it stops until it converges and then uh we TR on stage two and until we TR on stage n right so and in that case like all
the uh discrete uh steps uh discrete sentences are replaced by The Continuous SCE Vector so that's uh that's very simple kind of straightforward
training process yeah so what happens to uh to the to the results so here's like several example that I
can share with you that if you train the model this way then uh the model will behave very differently than the description of thoughts so we may test
this approach on synthetic data uh which means that you actually construct a graph uh between the uh starting noes and ending nodes right so basically
the graph has lots of like directed edges uh specify uh by the actual sentences right so here you
see that um so every lumus is a Yus so this basically means that there's a edge between the node lumpus to the node lpus so this actually con
basically this is actually give you the graph that you want to study and the answer is basically like a one of the node uh that can be connected uh
from the starting node right so the question is Alex grumpus or grumpus which means that is there an edge
between Alex and the Bumpus and is that edge between Alex and the Gus so that's a very simple kind of graph reasoning task uh in this case so U of
course like there's are ground true solutions by constructions but there's also a lot of solutions that is actually very misleading and may lead to
wrong answers right so because the graph is complicated so if you just run trend of ss the what happen is
that the CH of thoughts may make the mistake at the beginning right so once it make mistakes at the beginning by like checking okay if you go left
right so and you eventually we find you actually end with a node as a de end there's no way you can go all the way to the destination so in the Chann
of s case because it goes to the local Minima so it has to hallucinate an edge all the way to another
node of the graph so that it will continue to reach a destination which is wrong but in our case if you TR our models on this set and test on
different examples what you see here is that while uh the channel if the channel thought is short uh when it become longer and suddenly will give you
the correct path um uh from by the two hops because the optimal trajectory is to pass so it's actually very
interesting so people I actually we actually have some uh ideas about why it works right so and here's like some summary about like what's happening
inside this U chain of s uh Channel thoughts process right so for example like if you just do one continue Channel thoughts then you can also once you
have this Contin Channel s then you can ask the mod to decode on the next step about uh the next uh
next Noe that it may be able to decode right so then what you see is that the model seem to already know which uh node is already dead end because the
corresponding probability in the soft after soft Max already encoded implicit value functions whether it is actually dead end or not right so for
example here the uh therapist is actually dead end so the corresponding um softmax function actually
give you a very low number in term of probability right so this are that and other three they have pretty high probabilities so basically the model
after training actually learn this uh these structures so if you do a one step further then you have B basically two continu chn Source vectors here
and these two continu Source vectors after that and then you ask the mod to decode that particular
tokens and you will see that uh different tokens will have different probabilities and this probability actually really correspond to the connectivity
uh of whether these nodes will connect to the final outcome of the of the possible destinations right so these two are dead ends because there's no
Edge as going out so they are corresponding um soft Max values is actually very small right so but
there some know that actually has very high probability especially this one yeah so because it's very close to uh the actual uh Solutions so basically
it look like a in the Contin Channel s settings interestingly it encode all possible passes the intermediate like a latent uh and continu
representation actually incly incode all possible passes so that when uh you do the Sear search is actually
going through um all different partes until it hit the wrong it hit the right answer and then you can basically directly given the right answer you
find like what's the best path to go so this seem to be like a very efficient way of doing search right so if you think about how we do this in the
discret space then what happen is that the discr you have to do B search to basically encode all the
possible uh combinations and then you do like a uh branching and you do proving you do pruning until you find destination but in continu space look
like multiple possibilities can be super positioned together until you find the right answer then it will decode correctly that's something that's
last question so so here you have this probability of rompers and have this factorized this into uh like
a r o RP derive this yeah so sorry this one is this is because of the tokenizer we use so we are basically train on the to same tokenizer as llama so
tokenizer we basically breaking it into three pieces yeah okay and then you calculate that from this uh the second and last layer to get that yeah
yeah yeah exactly so basically like when you already consume this two Contin channel of thoughts and
also consume this eot and every token right and then you want to basically CH next token then what you do is you can basically plug in possible
outcomes uh and then check the probability right okay yeah and then you basically say if you have two sorts the probability of that would be higher uh
basically if you have two SS means that you actually do a two hop search right so you do two H search
then this probability actually contain all the possible uh outcomes and this outcome the probability look like they are created they are very created
with whether the future node will be connected to the end uh the destination that you want to go how do you know this two thoughts is actually
correspondent to two hops yeah so yeah so I think because each s when we when we do the when we do the
training we actually have every SCE represent a hop so basically this is by Design you train this using uh using this kind of structure then you know
each s correspond to one hop in this I see I see okay so this is more like kind of control experiment actually demonstrate what's the what's the
meaning of that yeah yeah exactly yeah okay got you yeah this is very interesting do you know why this
happens like there seems to be a notion of a value function or something somewhere because like where is this imported uh I think that's the
interesting part of this paper because we don't know yet uh it look like it's actually incode a value function but uh it's the value function yeah
it's interesting okay very interesting yeah it's it's kind of learn automatically from the training we don't
know we have no idea what is value function or we just train model like this way and then it give you this solutions I think might was to plug in this
NPC into it right because when you H have a NPC because you already got this uh correct trajectories yeah and then obviously you don't you don't let
them know but then let this MPC running underneath this thoughts um you know maybe on row this MPN
you can actually understand this Dynamics and how this going to um you know can generate the correct trory as you given I think it's say yeah it's
very interesting yes yeah so I think one interpretation uh is that uh it's possible that this latent space May incode some uh some super positional uh
structure of the graph right so basically if you think about like Quantum systems then uh any kind of
discretization is kind of measurement uh from the quantum uh super position right so by basically taking the product and then sample for it's exactly
same as doing like Quantum measurement right so but if you don't do measurement you just use the superposition state to do compute then the
superposition state may include all the possible passes at the same time uh by super position all the possible
passes together uh and then finally once you reach the destination then you just decode then maybe it's much more efficient then just do L just do
like a discrete token uh chain of thoughts yeah quite nice analogy yeah so that's one possibility but we don't know uh it's uh you want using a
Quantum formulation to really understand it but yeah so that's uh that's one possible explanation from when I
I saw this experiments I feel like this maybe one of reason but we don't know yeah we I see that's why you say I don't know do the super um U what the
super uh so basically you don't do the measurement therefore you don't generate tokens rather keep Sol running you it yeah interesting yeah that's one
possible explanation but we don't know that could be other explanation yeah okay so maybe uh yeah
sorry just one last thing I think I think maybe if we look at it from a value function perspective we might get some information and maybe these uh
soft mags are encoding some value information actually yeah yeah I think so it's it's definitely possible yeah okay so uh we actually check the
performance in the data set which is called the pros QA so it's basically a data set that I mentioned before
uh you construct a lot of graphs edes and then you ask the model whether the two nodes are connected or not so and we show that with like a longer and
longer thought thinking process and our approach getting better and better compared to channel thoughts because Channel thought will have a lot of
incorrect passes and um and also some of them they might have they will have hallucinations right so
that's the actually showing the strength of this approach of course this approach is more like experimental research purpose so U the performance on
gsm AK Is Not Great uh if you compare to um the general of source approach I mean you have fewer number tokens of course because we are using Chanel
Source but the accuracy is getting a little bit lower uh than uh the co but it's actually better than
all the nonot approaches right so of course that's uh on the synthetic data sets we actually see the pery is great uh compared to all the results
especially cross QA right so but in the for channel s cross QA uh performance is worse uh because there's a lot of like U hallucination happening and
a lot of dead end that ch s we're getting to uh it get confused uh but our approach is getting very well
it's doing things very well okay so of course there's a there's a a lot of there's some weakness of this paper so first of all the L tokens are not
interpretable people already ask these questions right so if you start with lat tokens and how what the hell is going on inside Network and maybe it's
not safe for humans and second is on test on AK for real data sets so we have actually another paper
that recently released which is called the token assorted and that paper is basically doing a mixture of uh continuous and discre tokens in order to
train the model better right so we all know that the regular coot they are basically doing text based tokens and then you use that to do training and
fine tuning but for uh our assoiated coot what we do here is that we actually convert some part of
the channel of sauce chunk into the latent discrete tokens so this will actually make the token make the CT shorter but at the same time also will
increase performance uh surprisingly so that's actually the idea right so how to compress the CT we can actually use vqe to do that right so given
this CT tokens we basically first like a chop it into chunks and then for each Chunk we basically use an
encoder to encode into latent vectors latent tokens and then you replace the coot with the latent tokens then you will have a you basically have like
a uh traces to Chanel and here's a performance right so we actually show that um with this shorter sequence and also the sequence that contain latent
tokens uh our approach actually give you better performance uh than other approaches including CT
it's better than coot in many kind of mass uh problems uh the G is a bit it's around like four to 5% right so in absolute numbers so know that all
these s models they are basically base models so it's not a fin uh on the mass so the performance a bit low uh if we find the mass I expect like a
even higher performance at the same time you see the co is getting shorter right so this is basically
length of Coots for each of the uh each of the models and you see that we actually can get an average of like a 177% uh dropping uh in term of the co
lens at the same time performance is getting better so it's actually very interesting so how we train it uh the training is actually is also very
straightforward uh with some simple tricks so here we start with CT and then we replace some of the CH
of CT into the latent vectors latent tokens right we are doing this AR called autograss replacement so we first randomly draw a a location of the C
and given the location every uh chunks of the C will be replaced by latent tokens and everything else after the location is keep fixed as a text based
l so then we have the prompt latent text by coot and solutions put all these four together and then
we train models accordingly it's very simple so and it's also fixing to uh it's fit very well into the existing autog training pipelines right so this
actually make the experiment pretty uh straightforward and we see that an hour approach is actually pretty working quite well compared to like a naive
replacement so you can EAS replace all the co chunks with uh with all the latent tokens but it
look like the model the pend model is not adapt to it very quickly so the performance actually drops potentially right so you can also do like a COR
replace right you can basically replace one at a time uh but this is actually pretty slow I mean actually our coconut training is pretty slow but our
mixture uh strategy is actually very efficient uh and it's very it also leads you the best
performance right so you can you can clearly see that this token assed paper is a combination of this coconut and also combination of the Dual formul
right so put them together and we actually end up with a better approach that can do both okay so this conclude the second option uh I'm not sure how
many time do I have I have like 1 hour right okay yeah so I think that should be the case and let's
do the uh option three uh I want to be a little bit faster here because there's a lot of mess so I want to only give you like a very rough idea about
what is going on here so traditionally we also think about like a symbolic structure and also neur Network structures as two uh tonomy two different
concept that cannot be compatible each with each other so sometime we think oh yeah we are using
symbolic structure we symbolic approach where NE approaches and uh they can be combined high level right so but here I want to show you some recent
progress that it's possible that you actually can see symbolic structures emerged from the deep learning trainings the train deep models and you kind
of exam its uh solution examine this solution and you realize that the solution is actually having
nice symbolic structures so that's actually something that uh you can uh it's kind of surprised for people to think about that maybe in the future uh
these two representation can actually be unified that's why we have this title called a tools unification of symbolic and neural uh decision making
right so this kind of like a thinking is actually related to uh kind of like a lasting debate about
how L model represent knowledge right so uh and there's one there basically two different uh School of thoughts right so one is basically saying oh
Lang model can do anything if you scale up the parameters so basically actually show emerging behaviors when the paramet become Higher and Higher and
more and more then you actually see everything suddenly works very well right so that's actually one
of the very optimistic thinking but on the other hand uh people also could ize the langage models performance saying that language model may be just
doing retrievals right so maybe we just like a remember all we have seen in the uh PR tring data sets and uh what we do is basically finding uh this
data uh in a very mechanic uh way right so um that's one way we can do mean there's a lot of evidences
right so recently there's a paper from Apple uh showing that if you do some you add some like unrelated sentences of all the uh prompt of the reading
task and all of a sudden the performance of all the reing model they drop substantially right so uh that's a that's a very surprising findings and
also May support the idea that a is still not robust and doing like a literally memorization so of
course uh we really want to uh basically like break this U to basically like get into the uh black box of large model to understand that better right
so that would be very nice to understand so for this particular product I'm basically using a very simple um reasoning task called modu addition so in
this modu addition task what happen is that you have two numbers A and B and you want to ask the
model to predict um uh the the result of additions right so predict C so both all A and B and C they are basically from uh this between zero and D
minus one so this kind of more modular addition so if you exceed that limit and you walk back uh to from from the beginning so if you um intuitively
what you see here is that if you Trend the new networks on this very simple task right so given a and b
you ask my network to predict C uh it will basically memorize this table right this called addition table as we have done in our childhood right so
you have increase table to memorize them and you do retrieval it get a correct answer but actually what it happens is that people already perceive
that if you ask the model to Trend this uh this problems and the learn representation is not one hard
representation it's often it's called FL basic representations right so you will actually see a lot of s or cosine uh in term of represent in the
representation space and uh basically the model is using the sign cosign to do addition to do to do addition for this particular tasks so this has
been supported by many papers right so here's like two papers I just put here uh one paper is from 202 and
245 I saw a very uh interesting paper which also give you the same kind of um knowledge so this thing sounded very surprising uh because uh people
usually think this is kind of intuitive right so it seems like it can use a different representation to do that so uh in this uh in our in this in my
particular like work uh so I'm trying to basically make this uh study as minimal as possible right so
here we only use like a two Network right two layer networks one is the lay networks have two layers so the bottom layer and the top layer the bottom
layer has um weights and they take this one hor representation A and B and then the top layer is basically like also one layer of weights and the
output some representation of c and the loss is very simple it's kind of two layer two loss but the loss
basically say you want to minimize the output of the two layer network uh and the distance between output and the one representation of the ground
true solution C in air to manner right so very simple uh approach and you actually what you actually see is that even with this very simple problem
setup uh once you train models uh you actually see very nice um solution structures uh from the uh
greenend trainings right so here is a graph that visualize the solutions so the x-axis is basically like the number of hidden uh node index and uh so
basically here you have 20 hidden nodes in the new networks and the y- axis is the frequency of each of the weights transformed into the FL basis
right so for each weights you may the weights will have like a uh some structur and can be visualized
more uh easily in the F basis so what you see is that indeed it learned F bases but also uh it actually show this very nice uh Global structures of
the solutions right so for example here in every frequency you actually see all the six Solutions which means that are frequency is covered by three
by four by Sorry by six different neurons so here every frequency is covered by four different neurons
right so for six different neuron so this called eror six Solutions you also can see that all the four Solutions so which means that some uh some uh
frequency is actually covered by like a four different neurons with some kind of like a flat um f kind of biosis there so you can see this two
different solutions one is called a six and the other called a four Solutions there right so and if you
scale this problem entire problem up right so from D equal to 7 to D equal to 33 you actually see like also very simple uh similar behaviors you get
this you basically in term of histograms you see these two kind of order of solutions it really happens uh so if you scale that to even higher uh
scale right so you have like a d equals like 127 which is like a larger can it should be supposed to be a
large table for additions right so what you see is you also see all the six and all the all the four Solutions there right so even if like this here
like the hidden neurons is like 512 which is a lot uh but for each frequency you only have uh four or six neurons representing it for the entire
Solutions so that's actually very interesting and uh we have a very uh kind of complicated U but very nice
uh mathematical formulations uh to uh to show uh the solution actually can be composed can from like a small and partial Solutions and this will
basically form the entire albic structure of the solution space right so in order to do that first of all you want to basically like decompose the A2
LW uh into the following form which is seems to be complicated but has a very nice structures that are
compatible to the weight uh weight structure and you also show that the weight structure is forming a semi- ring group SE sorry semi ring structure
structures and put them together what happen is that you can actually show that um I to show that like if you have a partial solution that satisfy
like one of the terms of the loss functions if have another part solution satisfies another part of the
solution structures then uh if you multiply these two together the multiplication is using the ring multiplication then this solution will basically
stify both of the constraints in the loss functions right so if you do this intuitively then you start with these Power Solutions and that only
satisfy that only will give you zeros in some term of the of the loss functions if you combine them
together then naturally you will end up with a solution that give you the global Optimizer for the entire MSC loss functions so this basically giv you
a way to construct the solution explicitly rather than rely on Gren scent to uh to give you the solution right so following this a you can actually
construct different solutions this include all the six solutions that you perceive all the four
solutions that you also see and also all the D Solutions that correspond to the perfect memorization right so uh perfect memorization basically means
that you are memorizing the entire addition table in the in the in the new networks and then you do retrievals right so but this all this solution is
actually High very high order so when doing the training the green sand would prefer to use a kind
of a lower order solution so what makes this kind of construction interesting is that if you actually check the Gren descent solution they actually
form they actually follows what the thetical construction look like right so first of all if you have you take all the GRE Solutions uh from the uh
from the experiments and then first of all you check whether they are order six and all four yes they
are order six and order four and second you check whether these solution are factorizable right so if you check the previous structure of solutions
they are basically solutions that is composed from smaller order Solutions using the ring multiplication and the ring addition right so and you
actually see that in the green ex Solutions these Solutions obtained by the empirical approach is also
following the same factorization structures showing that this analytical form also play an important role uh in determining the Solutions in empirical
sense right so facturation error is small and finally and what you see here is that the solution not only factorized but they factorized into the form
that is constructed in the analytical formula so that's actually very interesting and this might
give you the idea that maybe in the future you don't need to use green descent right so you may there may exist another Alaric operation uh so that
you can construct a solution that is similar to what great s can achieve right so in that case like the solution that obtained by Green scent is
actually has symbolic representation uh we don't know but we might know later so maybe in the future there
will be new approaches uh to optimize new networks uh so that the representation become very useful right so this actually conclude the talk uh maybe
in the future we instead of like praying the machines so that the machine can give you the wrong the right solution without too much issues we might
open the black box of new networks and actually come to another a different stage of uh of training
and we actually know what's going on inside so thank everyone for the attention and let me know if there are more questions thanks Dr yeah thank you
yeah go ahead thank you can you hear me yeah uh so I want so this the phenomenon is observed that they're using the such a specific structure to solve
this module addition tasks yeah in the training center where the only training data is this like
additional task I wonder like in the real case like in a complicated pre-training LM uh there the math task is also train with other like Linguistics
task like is the same observation happen again or they doing yeah so I think the two papers that you I actually uh share right so they actually Trend
they basically like just directly check the pr Trend models people provided so it's basically Trend
with all the datas so it's not only the s data the addition data I see I see thank you yeah so if you check these two papers right so uh the P
language models use FL features to comput addition so this is basically they are basically checking the Lama models right so um yeah so these two are
basically the papers that you can check yeah thank you okay do we know why gradient descent favors this
modular factorization structure yeah I think first of all I there some hidden slides unfortun I didn't put this into this version uh so you can
actually show that uh the two gradient basically the two global solutions they are connected uh by a pass of all zero uh losses for example if you
have solution Z that is already a global solution for the objectives you have another solution which is z
times uh something which is also Global Solutions uh by the property of the uh semi- ring structure uh then you can prove that these two solutions
actually connected by a pass and U if you apply weight Decay regularization then the weight Decay will favor simpler Solutions I see okay yeah so
that's a that's an interesting phenomenon of course it does not give you the full picture because the four
picture may be the case that you have two solution that not connected then yeah but the at least they give you some kind of idea because the is like a
lower order yeah right and this holds also for Adam or just like radient with weight Decay like if you add momentum you will also have this same low
complexity favor yeah actually all the experiments here are done by Adam so this is Adam oh I meant
theoretically though yeah but okay yes theoretically yes because uh in uh in the paper I don't really analyze the Dynamics we basically analyze like
what's happening after we go into the Optimal Solutions right so it basically apply to Adam as well okay got it thank you okay sorry maybe one last
question from the
audience okay if not thanks uh Dr again and uh we we're going to come back uh tomorrow and uh then we will focus bit more on this agent and Dr T going
to chair the session um we will uh we'll see you see you guys tomorrow then bye thank youe