[libre-riscv-dev] Scoreboard vs Tomasulo

[libre-riscv-dev] Scoreboard vs Tomasulo

[Yehowshua]

This website gives an excellent comparison

https://www.cs.umd.edu/~meesh/cmsc411/website/projects/dynamic/intro.html <https://www.cs.umd.edu/~meesh/cmsc411/website/projects/dynamic/intro.html>

In general, it sits that Tomasulo is a distributed scoreboard.

The biggest drawback of Tomasulo is the central data bus as the messaging scheme. One could use a mesh NOC between functional units.


Given that we have well over 15 functional units, I would have thought a distributed scoreboard instead of a centralized one makes more sense.

Of course, I have no numbers to back this up. But these are just some thoughts.


I know we’re using a ring/circular buffer for messaging at the moment.
Given that we have hard deadlines and limited resources, we can stick with this. But these are questions I do wonder about the answer to. I wonder if there are any papers where this has been explored at sufficient depth to draw conclusions.

Yehowshua
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

On Saturday, May 16, 2020, Yehowshua <yimmanuel3@gatech.edu> wrote:

> This website gives an excellent comparison
>
> https://www.cs.umd.edu/~meesh/cmsc411/website/projects/dynamic/intro.html
> <https://www.cs.umd.edu/~meesh/cmsc411/website/projects/dynamic/intro.html
> >


ah good find.  particularly because, like all Academic literature on the
6600, it is factually partially correct but at the same time dangerously
misleading and factually wrong.  this for example is wrong:

However, the scoreboard is limited in that it does not handle WAR and WAW
hazards very well.

the original 6600 handles WAR extremely well, only stalling on WaW
condition, detection which did not matter greatly because the pipelines
were only at most 2 stages long anyway (Mitch only noticed after rereading
last year that the FP ADD of the 6600 was 2 stage pipelined.  no academic
literature has acknowledged or noticed this).

you have to understand that the majority of designs at the time were around
10 to 12 clocks per instruction.  the CDC6600 got that down to FOUR and
consequently represented a 2 to 3 fold increase in performance.



> In general, it says that Tomasulo is a distributed scoreboard.


mmm no.  the ROB, which is the key component, is centralised, and the CDB
is a central resource.



> The biggest drawback of Tomasulo is the central data bus as the messaging
> scheme. One could use a mesh NOC between functional units.


ah, not quite the biggest: as you correctly say, all you need is multiple
CDBs and that problem is solved.

no, the biggest draw of Tomasulo is that the ROB (reorder buffer) numbering
is in *binary*, and bear in mind that the ROB is a CAM, the larger that
gets the more power it takes.

let us say we have a 32 entry ROB, let us say we have 5 bits for the ROB
CAM key.

this not only means that when trying to put a result into the CAM we have
5x32 XOR gates to fire - on every cycle - if we want 2 Common Data Buses we
now need DOUBLE that...

... *and we need clash detection*.

by contrast, 6600 scoreboards have *unary* numbering and consequently the
CAM is degenerate and becomes a single AND gate activation.

not only that but if you wish to check or activate more than one register
simultaneously then that... is... just... more than one corresponding AND
gate activating at the CAM.

perfectly simple, very little power, perfect for dropping multi issue on
top.




> Given that we have well over 15 functional units, I would have thought a
> distributed scoreboard instead of a centralized one makes more sense.


the document, whilst useful as a starting point, has misled you on two
counts (so far)

the key is that binary numbering requires power sucking CAMs.  hence why
the 6600 scheme, augmented to precise, and even the original 6600 Q Table
numbering has been converted from binary to unary.

not Tomasulo.

actually.... in 6600, the DMs are still centralised, it is however the case
that the unary numbering results implicitly in opportunities for
parallelism (see below) and consequently could be termed "distributed".

also, the role of the Function Units, the GORD/REQ latching, that is done
as part of the critical acknowledgement and communication *with* the
scoreboards, and is done very close to the pipelines, *not* in the DMs
themselves, so in that regard, yes it is "distributed".

Function Units are equivalent to "Reservation Station Rows" from Tomasulo
terminology.  the multiple rows per Tomasulo Reservation Station *also*
requires that those be CAMs!

yet more power-sucking!

given that Intel processors use Tomasulo, we start to see why Intel
processors suck so much power.


> Of course, I have no numbers to back this up. But these are just some
> thoughts.
>
>
> I know we’re using a ring/circular buffer for messaging at the moment.


yes for the future version.

for the simple 180nm version it will be simple direct regfile port
broadcast buses, connected one to one with the corresponding Function Unit
Operand input.

thus, FU operand 1 will be directly connected to Regfile Port1 Broadcast
Bus.

FU operand 2 - if there *is* an operand 2 - will connect to Regfile Port2
Broadcast Bus.

*if we have time* then we can drop in the cyclic buffers, and when data
comes out of Regfile Port2 it is cyclically shifted to Op... 1 or Op3 or
whatever is required.



> Given that we have hard deadlines and limited resources, we can stick with
> this. But these are questions I do wonder about the answer to. I wonder if
> there are any papers where this has been explored at sufficient depth to
> draw conclusions.


i encountered when i investigated Tomasulo a way to do multi issue in an
academic paper.

the method, which involved stratifying the Reorder Buffer into 4 separate
slices, was awful.

the binary numbering on the ROB causes massive headaches because it
required special queues of binary ROB indices to represent the multi issue
requests.

when converted to unary, multiple bits may be set to indicate REG1 REG5
REG7 *in one cycle* on the *same wires* because one wire is dedicated to
each reg.

this is precisely and exactly what you need for multi issue and it is so
laughably simple.

l .


-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

https://www.cs.umd.edu/~meesh/cmsc411/website/projects/dynamic/example1.html

we have converted Fi Fj Fk and Rk Rj and Qk Qj to UNARY.

bit level.

this is how we can push multiple entries per clock (implement multi issue)
into the DMs.

in the original 6600 reg numbers were sometimes unary and sonetimes binary,
with encoder-decoders converting between them.

this reduced gate count at the cost of making the encoders and decoders a
one data per clock bottleneck.

by preserving everything as unary we can transfer up to the full number of
unary bits as hazard information, in one cycle.

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

https://www.cs.umd.edu/~meesh/cmsc411/website/projects/dynamic/tomasulo.html

register renaming to eliminate output and anti-dependences, i.e. WAW and
WAR hazards

also misleading in that it completely misses that the 6600 Q-Tables also
provide the concept of register renaming.

the Latches at the Function Unit are directly equivalent to RS Row latches
and are consequently "nameless registers".

hence you start to see why the Academic literature is hopelessly misleading
and why Mitch was pretty much compelled to write those two book chapters.

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

Tomasulo's algorithm implements register renaming through the use of what
are called reservation stations.

see? total failure to mention the 6600 Computation Unit latches as being
directly equivalent to that.

therefore total systematic failure in the academic literature *world wide*
to mention that the 6600 had and was the world's first computer design to
have register renaming.


On Saturday, May 16, 2020, Luke Kenneth Casson Leighton <lkcl@lkcl.net>
wrote:

> https://www.cs.umd.edu/~meesh/cmsc411/website/projects/
> dynamic/tomasulo.html
>
> register renaming to eliminate output and anti-dependences, i.e. WAW and
> WAR hazards
>
> also misleading in that it completely misses that the 6600 Q-Tables also
> provide the concept of register renaming.
>
> the Latches at the Function Unit are directly equivalent to RS Row latches
> and are consequently "nameless registers".
>
> hence you start to see why the Academic literature is hopelessly
> misleading and why Mitch was pretty much compelled to write those two book
> chapters.
>
> l.
>
>
>
> --
> ---
> crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
>
>

-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

https://www.cs.umd.edu/~meesh/cmsc411/website/projects/dynamic/tomasulo.html

Thus, unlike in scoreboarding where the functional unit would stall during
a WAR hazard, the functional unit is free to execute another instruction

idiot.

utterly false and misleading.

i am too irritated by these repeated glaring errors and misunderstandings
right now to go into it in detail.

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

 In Tomasulo's algorithm, the control logic is distributed among the
reservation stations, whereas in scoreboarding, the scoreboard keeps track
of everything.

false and misleading.

as previously mentioned, the Computation Unit latches (equivalent to RS
Row) contain READ REQUEST and GOREAD synchronisation that effectively is
just as "distributed" as Tomasulo.

also as previously mentioned the ROB CAM is just as centralised as the DMs.

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

https://www.cs.umd.edu/~meesh/cmsc411/website/projects/dynamic/modern.html

this entire section is misleading in several ways. it's not wrong per se,
it's just not the only solution.

the *other* solution is to have "precise shadowing".

this is a system (described at the end of Mitch 2nd chapter) which allows
result *computation* to run ahead but where result *commit* is prevented
and prohibited.

(the gate count required to do so is extremely low.)

this on the face of it would seem to stop FUs from progressing because they
cannot do so until they receive their operands... and um how could they do
that if writing to the regfile is prohibited?

the answer is very simple: that is what Operand Forwarding is for, aka
"Common Data Bus(es)".

so by preventing write commit, we can run ahead for a considerable time, no
stalling at all.

when the branch prediction resolves, we then either:

* release the shadow which allows a ton of results to begin committing.
order is not strictly relevant except as preserved by the hazard detection

* call the "go die" line which drops absolutely every downstream result and
partial computation underway on the floor.  the PC then issues ops down the
alternative path.

(i did actually try dual path execution: theoretically our engine can do
it, i just could not get it to work last year, it was taking too long to
debug, and i had to focus on other things).


the thing that is particularly misleading about this particular section is
that it instills in the reader's mind the false impression that there is a
mandatory correlation between speculative execution and operand order
committing.

this is just false.

if the commit can be stalled until it is guaranteed known that, after that
point it is perfectly *safe* to commit (the operation is 100% going to
succeed)

    AND

if there are no hazards remaining

   THEN

who _cares_ what order results are committed? answer: nobody.



so... useful inasmuch as it is written by an academic and thus is really
readable.

sadly though it contains the usual glaring errors, misunderstandings and
omissions which are widely pervasive and common throughout the *entire*
academic world, even by those who claim to have studied 6600 scoreboards,
even by those who claim to be experts on RISC design.

for the most part that involves cursory reading the patent on 6600 Q
Tables, *not* studying Thornton's book in great depth and taking the time
to translate the (now archaic) transistor level diagrams into modern IEEE
standard electronics symbols - something that Mitch went to a lot of
trouble to do.

i trust that this gives you something of an eyeopener into the extent of
the misapprehensions that the Academic community is under, regarding out of
order engine design.

the closest working implementation i have seen similar to what we are doing
is a conversion of an existing RV32 design to superscalar and multi issue
by a chinese guy who has not revealed his name, online.

he also did multi-bit unary transitive relationships and dependency
tracking in exactly the same way that we will, having re-derived 6600
scoreboards with no reference to any academic literature.

however he did not at the same time increase load store data bandwidth, and
on top of RV32I it is a hell of a lot of extra work to bring up to where we
can use it... and it's all in verilog.

l.




-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Yehowshua]

This is a very intricate and complicated subject matter for sure.

At some point, it be great to really break things down and make them more accessible.

Yehowshua
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

On Saturday, May 16, 2020, Yehowshua <yimmanuel3@gatech.edu> wrote:

> This is a very intricate and complicated subject matter for sure.


yes, except it doesn't have to be.  the actual
https://en.wikipedia.org/wiki/Levenshtein_distance between Tomasulo and
6600 really is not that great.

i thought it would be fun to use a new unpronounceable word i learned
yesterday :)


> At some point, it be great to really break things down and make them more
> accessible.


yes. it comes down to time.

start with this.

1. Begin from Tomasulo.  neither TS nor original 6600 have precise
exceptions so we leave that out for now.

2. Start by only allowing one row per Reservation Station.

3. Expand the number of RSes so that if you were to count the total number
of places operands are stored, they are the same.

(another way to put this is, "flatten all 2D RSes into 1D")

4. where pipelines were formerly connected exclusively to one RS,
*preserve* those connections even though the rows are now 1D flattened.

(another way to put this is: we have a global 1D naming scheme to reference
the *operand latches* rather than a 2D scheme involving RS number in 1
dimension and the row number in the 2nd)

5. give this 1D flattening an UNARY numbering scheme.

6. make the size of the Reorder Buffer EXACTLY equal to the number of 1D
flattened RSes.

7. rename RSes to "Function Units" (actually in Thornton's book the phrase
"Computation Units" is used)

thus, at this point in the transformation, the ROB row number *IS* the
Function Unit Number, the need to actually store the ROB # in the
Reservation Station Row is REMOVED, and consequently the Reservation
Stations are NO LONGER A CAM.

8. give all register file numbers (INT FP) an UNARY numbering.

this means that in the ROB, the CAM, which has to look up the register
number by hitting the CAM on every cycle, now only has to match a single
AND gate.

bitvectors therefore replace CAMs.

with the ROB now having rows of bitvectors, it is now termed a "Matrix".

the left side of the ROB, which used to contain the RS Number in unary, now
contains a *bitvector* Directed Acyclic Graph of the FU to FU dependencies,
and is split out into its own Matrix.

this we call the FU-FU Dependency Matrix.

the remainder of the "ROB" contains the register numbers in unary Matrix
form, and with each row being directly associated with a Function Unit, we
now have an association between FU and Regs which preserves the knowledge
of what instruction required which registers, *and* who will produce the
result.

this we call the FU-Regs Dependency Matrix.

that *really is it*.

take some time to absorb the transformation which not only preserves
absolutely every functional aspect of the Tomasulo Algorithm, it
drastically simplifies the implementation, reduces gate count, reduces
power consumption *and* provides a strong foundation for doing arbitrary
multi-issue execution with only an O(N) linear increase in gate count to do
so.


further hilariously simple additional transformations occur to replace
former massive resource constrained bottlenecks, due to the binary
numbering on both ROB numbers and Reg numbers, with simple large unary NOR
gates:

* the determination of when hazards are clear, on a per register basis, is
a laughably trivial NOR gate across all columns of the FU-REGs matrix,
producing a row bitvector for each read register and each write register.

* the determination of when a Function Unit may proceed is a laughably
trivial NOR gate across all *rows* of the *FU-FU* Matrix, producing a
row-based vector, determining that it is "readable" if there exists no
write hazard and "writable" if there exists no read hazard.

* the Tomasulo Common Data Bus, formerly being a single chokepoint
binary-addressing global Bus, may now be upgraded to *MULTIPLE* Common Data
Buses that, because the addressing information about registers is now in
unary, is likewise laughably trivial to use cascading Priority Pickers (a
nmigen PriorityEncoder and Decoder, back-to-back) to determine which
Function Unit shall be granted access to which CDB in order to receive (or
send) its operand (or result).

* multi-issue as i mentioned a few times is an equally laughably trivial
matter of transitively cascading the Register Dependency Hazards (both read
and write) across future instructions in the same multi issue execution
window. instr2 has instr1 AND instr2's hazards.  instr3 has instr1 AND
instr2 AND instr3's hazards and so on.  this just leaves the necessity of
increasing register port numbers, number of CDBs, and LD/ST memory
bandwidth to compensate and cope with the additional resource demands that
will now occur.

the latter is particularly why we have a design that, ultimately, we could
take on ARM, Intel, and AMD.

there is no reason technically why we could not do a 4, 6 or 8 multi issue
system, and with enough Function Units and the cyclic buffer system (so as
not to require a full crossbar at the Common Data Buses), and proper
stratification and design of the register files, massive Vector parallelism
at the pipelines would be kept fully occupied without an overwhelming
increase in gates or power consumption that would normally be expected, and
scalar performance would be similarly high as well.

l.




-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Jeremy Singher]

> However, the scoreboard is limited in that it does not handle WAR and WAW
> hazards very well.

> the original 6600 handles WAR extremely well, only stalling on WaW
> condition, detection which did not matter greatly because the pipelines
> were only at most 2 stages long anyway (Mitch only noticed after rereading
> last year that the FP ADD of the 6600 was 2 stage pipelined.  no academic
> literature has acknowledged or noticed this).

I would argue that the CDC6600 handles WAR better than an in-order
machine, but not better than a machine with full renaming. In the
CDC6600, the scoreboard would block the completion of the W inst until
the R inst has read its operands. In Tomasulo, the W inst can proceed
immediately before the R inst is ready, since the destination register
is physically renamed.

>  ROB is a CAM,

What? This is not true. Instructions in flight track their index in
the ROB, at completion, they write success bit according to the index
they have. Not a CAM. Simple addressed write.

> given that Intel processors use Tomasulo, we start to see why Intel
> processors suck so much power.

Power consumption on a processor is primarily from high switching
rates on large register files, and SRAM leakage. The scheduling logic
is a miniscule portion. Intel cores are power-hungry because
historically they have targeted the desktop/workstation market, and do
not have as many decades of experience as other firms optimizing for
power-constrained environments.

Also almost all modern OOO cores use Tomasulo because Tomasulo is
better at avoiding false hazards compared to scoreboarding. Tomasulo
also makes it easier to implement precise traps, something which is
notoriously challenging  with just a scoreboard.

Lets be clear here. The CDC6600 AVOIDS WAR and WAW hazards by BLOCKING
younger instructions (See pg3 in Mitch Alsup's text). Tomasulo's
supports OOO execution PAST WAR and WAW hazards, enabling the core to
exploit more ILP across its functional units.

Luke, I agree with you that the reservation station rows in Tomasulo's
can be equivalent to the operand latches in Scoreboarding. However,
adding many operand-latches (by having many functional units) does not
scale as nicely as just adding a few more rows to Tomasulo's
reservation stations. Especially since designing the CAMs for
Tomasulo's is a well studied, and well-understood problem

> 3. Expand the number of RSes so that if you were to count the total number
> of places operands are stored, they are the same.

> (another way to put this is, "flatten all 2D RSes into 1D")

This does not scale well. A 2D CAM is far more area efficient than a
1D flattened vector.

> 7. rename RSes to "Function Units" (actually in Thornton's book the phrase
> "Computation Units" is used)

So a Tomasulo's design with 4 rows now becomes a "flattened" design
with 4 functional units? The additional cost of the functional units
is enormous.

On Sat, May 16, 2020 at 10:33 AM Luke Kenneth Casson Leighton
<lkcl@lkcl.net> wrote:
>
> On Saturday, May 16, 2020, Yehowshua <yimmanuel3@gatech.edu> wrote:
>
> > This is a very intricate and complicated subject matter for sure.
>
>
> yes, except it doesn't have to be.  the actual
> https://en.wikipedia.org/wiki/Levenshtein_distance between Tomasulo and
> 6600 really is not that great.
>
> i thought it would be fun to use a new unpronounceable word i learned
> yesterday :)
>
>
> > At some point, it be great to really break things down and make them more
> > accessible.
>
>
> yes. it comes down to time.
>
> start with this.
>
> 1. Begin from Tomasulo.  neither TS nor original 6600 have precise
> exceptions so we leave that out for now.
>
> 2. Start by only allowing one row per Reservation Station.
>
> 3. Expand the number of RSes so that if you were to count the total number
> of places operands are stored, they are the same.
>
> (another way to put this is, "flatten all 2D RSes into 1D")
>
> 4. where pipelines were formerly connected exclusively to one RS,
> *preserve* those connections even though the rows are now 1D flattened.
>
> (another way to put this is: we have a global 1D naming scheme to reference
> the *operand latches* rather than a 2D scheme involving RS number in 1
> dimension and the row number in the 2nd)
>
> 5. give this 1D flattening an UNARY numbering scheme.
>
> 6. make the size of the Reorder Buffer EXACTLY equal to the number of 1D
> flattened RSes.
>
> 7. rename RSes to "Function Units" (actually in Thornton's book the phrase
> "Computation Units" is used)
>
> thus, at this point in the transformation, the ROB row number *IS* the
> Function Unit Number, the need to actually store the ROB # in the
> Reservation Station Row is REMOVED, and consequently the Reservation
> Stations are NO LONGER A CAM.
>
> 8. give all register file numbers (INT FP) an UNARY numbering.
>
> this means that in the ROB, the CAM, which has to look up the register
> number by hitting the CAM on every cycle, now only has to match a single
> AND gate.
>
> bitvectors therefore replace CAMs.
>
> with the ROB now having rows of bitvectors, it is now termed a "Matrix".
>
> the left side of the ROB, which used to contain the RS Number in unary, now
> contains a *bitvector* Directed Acyclic Graph of the FU to FU dependencies,
> and is split out into its own Matrix.
>
> this we call the FU-FU Dependency Matrix.
>
> the remainder of the "ROB" contains the register numbers in unary Matrix
> form, and with each row being directly associated with a Function Unit, we
> now have an association between FU and Regs which preserves the knowledge
> of what instruction required which registers, *and* who will produce the
> result.
>
> this we call the FU-Regs Dependency Matrix.
>
> that *really is it*.
>
> take some time to absorb the transformation which not only preserves
> absolutely every functional aspect of the Tomasulo Algorithm, it
> drastically simplifies the implementation, reduces gate count, reduces
> power consumption *and* provides a strong foundation for doing arbitrary
> multi-issue execution with only an O(N) linear increase in gate count to do
> so.
>
>
> further hilariously simple additional transformations occur to replace
> former massive resource constrained bottlenecks, due to the binary
> numbering on both ROB numbers and Reg numbers, with simple large unary NOR
> gates:
>
> * the determination of when hazards are clear, on a per register basis, is
> a laughably trivial NOR gate across all columns of the FU-REGs matrix,
> producing a row bitvector for each read register and each write register.
>
> * the determination of when a Function Unit may proceed is a laughably
> trivial NOR gate across all *rows* of the *FU-FU* Matrix, producing a
> row-based vector, determining that it is "readable" if there exists no
> write hazard and "writable" if there exists no read hazard.
>
> * the Tomasulo Common Data Bus, formerly being a single chokepoint
> binary-addressing global Bus, may now be upgraded to *MULTIPLE* Common Data
> Buses that, because the addressing information about registers is now in
> unary, is likewise laughably trivial to use cascading Priority Pickers (a
> nmigen PriorityEncoder and Decoder, back-to-back) to determine which
> Function Unit shall be granted access to which CDB in order to receive (or
> send) its operand (or result).
>
> * multi-issue as i mentioned a few times is an equally laughably trivial
> matter of transitively cascading the Register Dependency Hazards (both read
> and write) across future instructions in the same multi issue execution
> window. instr2 has instr1 AND instr2's hazards.  instr3 has instr1 AND
> instr2 AND instr3's hazards and so on.  this just leaves the necessity of
> increasing register port numbers, number of CDBs, and LD/ST memory
> bandwidth to compensate and cope with the additional resource demands that
> will now occur.
>
> the latter is particularly why we have a design that, ultimately, we could
> take on ARM, Intel, and AMD.
>
> there is no reason technically why we could not do a 4, 6 or 8 multi issue
> system, and with enough Function Units and the cyclic buffer system (so as
> not to require a full crossbar at the Common Data Buses), and proper
> stratification and design of the register files, massive Vector parallelism
> at the pipelines would be kept fully occupied without an overwhelming
> increase in gates or power consumption that would normally be expected, and
> scalar performance would be similarly high as well.
>
> l.
>
>
>
>
> --
> ---
> crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
> _______________________________________________
> libre-riscv-dev mailing list
> libre-riscv-dev@lists.libre-riscv.org
> http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

On Sat, May 16, 2020 at 9:16 PM Jeremy Singher <thejsingher@gmail.com> wrote:

> > However, the scoreboard is limited in that it does not handle WAR and WAW
> > hazards very well.
>
> > the original 6600 handles WAR extremely well, only stalling on WaW
> > condition, detection which did not matter greatly because the pipelines
> > were only at most 2 stages long anyway (Mitch only noticed after rereading
> > last year that the FP ADD of the 6600 was 2 stage pipelined.  no academic
> > literature has acknowledged or noticed this).
>
> I would argue that the CDC6600 handles WAR better than an in-order
> machine, but not better than a machine with full renaming.

Jeremy: 6600 designs *do* have register "renaming".  i said already,
a number of times.  and if you have studied and listened to the Academic
literature - and believe it - you are misinformed by that literature, and will
continue to be misinformed by that literature.

please understand: i have spent over nine months studying this, learning
from Mitch Alsup, one of the world's leading high-performance computing
architects, implementing this and it's not going to get chucked out to
implement something that takes six MONTHS to fully understand is a
better design, to be replaced by something that would take another six
months to implement *and we then could not do multi-issue on top of it*.

please therefore take the time to understand this design, particularly when
we are right in the middle of a time-critical deadline.

> In the
> CDC6600, the scoreboard would block the completion of the W inst until
> the R inst has read its operands. In Tomasulo, the W inst can proceed
> immediately before the R inst is ready, since the destination register
> is physically renamed.

Jeremy: the "renaming" carried out by the 6600 does exactly this same
job.

Academics unfortunately have *only* studied the *patent* - not Thornton's
book nor read Mitch Alsup's book chapters which he wrote based on 30+
years experience of implementing 6600-style execution in the 88100,
AMD K9 and AMD Opteron Series of CPUs.

the 6600 patent *ONLY* covers the Q-Tables.

the description of the Q-Tables does *NOT* include the Computation Units
where the operand latches are.

these latches *ARE* the "renamed registers" (actually, nameless registers).

> >  ROB is a CAM,
>
> What? This is not true. Instructions in flight track their index in
> the ROB, at completion, they write success bit according to the index
> they have. Not a CAM. Simple addressed write.

ah appreciated the correction.  i knew there were CAMs somewhere.
when i discovered that, i dropped further investigation.

i am however aware that if you wish to do multi-issue, the ROB will need
to be multi-ported SRAM.  without multi-porting the SRAM, you simply
cannot write multiple entries.

this is where the problems (read: nightmares) for Tomasulo start to manifest,
and progress from there through multiple Common Data Bus channels into
multi-ported (NxR, NxW) CAMs at the Reservation Station level.

i've been through this, ok, over 18 months ago.

by contrast, when converted to single-bit unary vectors, each entry in the
Matrices (covered by a single SR NAND latch) is *independent*.  multiple
of those may be set in a single cycle.

there has been a *massive* amount of thought gone into this, ok?


> > given that Intel processors use Tomasulo, we start to see why Intel
> > processors suck so much power.
>
> Power consumption on a processor is primarily from high switching
> rates on large register files, and SRAM leakage.

given that we're planning around 28 Function Units and have a 128-entry
register file, the calculations that i did for that would put the Dependency
Matrices around a whopping 50,000 gates, using SR NAND Latches.

(if we did it as DFFs it would be an unacceptable 250,000 gates)

> The scheduling logic is a miniscule portion.

not for the large-scale out-of-order systems that we are planning to
do, it's not,
due to the large sized GPU register file (128 entries.  actually 2x
128 - 128 FP,
128 INT).

we may have to *down*-scale via a regfile cache to cope with that.
discussion for another time.


> Intel cores are power-hungry because
> historically they have targeted the desktop/workstation market, and do
> not have as many decades of experience as other firms optimizing for
> power-constrained environments.

appreciated the insight.

> Also almost all modern OOO cores use Tomasulo because Tomasulo is
> better at avoiding false hazards compared to scoreboarding.

in the versions of scoreboards implemented by academics - because they
fail to comprehend them properly - because they only read the patent
on Q-Tables - yes.

in the version that Mitch Alsup - a world leading expert on Scoreboard
Mechanics - taught me:

no.

also, the IIT Madras RISE Group Shakti E-Class processor is *also* a
scoreboard design.  this was designed "from first principles" by Professor
Kamatoti, and it was only through learning from Mitch that i was able
to recognise what he had done as being identical to a 6600 scoreboard.

> Tomasulo
> also makes it easier to implement precise traps, something which is
> notoriously challenging  with just a scoreboard.

no it is not.  this is *yet again* complete factual misinformation and ignorance
from Academic literature.

can i ask: have you read Patterson's book?  this is the usual place where
most people get their misinformation from.  the release of this misinformation
was why Mitch Alsup - a very pissed-off and exasperated Mitch - wrote those
two chapters, back around... 2008 (i think).

precise exceptions, traps, interrupts, branch speculation and predication may
all be achieved through "shadowing".  see the latter part of the 2nd chapter of
Mitch's book.

see the section "Shadowing" in this page:
https://libre-soc.org/3d_gpu/architecture/6600scoreboard/

it's very simple: you hold write until such time as the opportunity for "damage"
is cleared.  Operand Forwarding allows instructions to run ahead, using results
that would otherwise have to be received only first by going through
the Regfile.

*this is already implemented* and the unit tests demonstrated it as
*fully functional*


> Lets be clear here. The CDC6600 AVOIDS WAR and WAW hazards by BLOCKING
> younger instructions (See pg3 in Mitch Alsup's text). Tomasulo's
> supports OOO execution PAST WAR and WAW hazards, enabling the core to
> exploit more ILP across its functional units.

this is only because the 1965 CDC6600 did not have an Operand Forwarding Bus
(instead it used the regfile as a same-clock forwarding bus, by reading on one
edge and writing on the other.  this unfortunately meant that without an OpFwd
Bus, stall had to occur).

this DOES NOT INHERENTLY MEAN that **ALL** Scoreboards are rubbish.

if you add an Operand Forwarding Bus (equivalent to the CDB in Tomasulo)
the exact same parity functionality is achieved (without the severe downsides).


> Luke, I agree with you that the reservation station rows in Tomasulo's
> can be equivalent to the operand latches in Scoreboarding. However,
> adding many operand-latches (by having many functional units) does not
> scale as nicely as just adding a few more rows to Tomasulo's
> reservation stations.

this is incorrect.  you may be under the mistaken impression that the pipelines
duplicated as well.  this clearly would be inadviseable because the FU's job
is to serve one and only one result.  therefore you know that this is a mistaken
understanding, and that consequently you have multiple FUs for the same
pipeline.  (termed "Concurrent Computation Unit").

therefore it's minimal.  excluding the latches themselves (the rows
have latches, the
CompUnits - aka FunctionUnits have latches, the total number of latches after
flattening is the same), the number of gates added by each CompUnit is
minimal: around... 80, maximum.

that's peanuts.

> Especially since designing the CAMs for
> Tomasulo's is a well studied, and well-understood problem

the key is the combination of those two:

A) well-understood... *and*
B) a problem.

have a look at how multi-issue is done using Tomasulo.  it's shockingly bad.


> > 3. Expand the number of RSes so that if you were to count the total number
> > of places operands are stored, they are the same.
>
> > (another way to put this is, "flatten all 2D RSes into 1D")
>
> This does not scale well. A 2D CAM is far more area efficient than a
> 1D flattened vector.

a 1D bitvector.  a single, 1D bitvector.  of 32 bits in length.

compared to 32 5-bit binary addresses.

just the latches alone, that math does not stack up.

now let's take it to 2D.

let's say that we have 10 FUs and 32  Regs.

in Tomasulo that is a 5-bit address.  10x5 = 50 DFF latches.  DFF
latches are 10 gates so that's 500 gates.

in 6600 that is 10x32 = 320 SR NAND Latches.  SR NAND latches are 2
gates.  320x2 is 640 gates.

not very different in terms of latch count, is it?


now let's do the address-checking.

to check a register as "active" - bear in mind that this is per
register and consequently
if doing multi-issue that's a batch of address-checking gates *per register*.

for Tomasulu, this requires 10x5 = 500 XOR gates (plus some
multi-input AND gates on each row)   XOR is 4 gates, so that's 2,000
gates *PER MULTI-ISSUE LANE*.

for the 6600, that's 10x32 **AND** gates.  320 AND gates.

and of those 320 AND gates, they are re-useable for multi-issue
WITHOUT REQUIRING ADDITIONAL GATES.

so that is 320 AND gates for single-issue
that is zero extra gates for dual-issue
that is zero extra gates for quad-issue
zero extra gates for octal-issue.

start to make sense now, why unary encoding as 1D vectors is less
gates and less power than binary CAMs?

if you want to check multiple unary bits (a critical prerequisite for
multi-issue), all you do is... set... multiple... bits.

what happens?  multiple AND gates fire.  in the same clock.   and
that's exactly what we want.


> > 7. rename RSes to "Function Units" (actually in Thornton's book the phrase
> > "Computation Units" is used)
>
> So a Tomasulo's design with 4 rows now becomes a "flattened" design
> with 4 functional units? The additional cost of the functional units
> is enormous.

no, it's 4 x 80 gates (approx).  that's not 4x duplication of ALU
pipelines *behind* those 4 Function Units.  see Mitch's 2nd Chapter,
"Concurrent Computation Unit"

https://libre-soc.org/3d_gpu/architecture/6600scoreboard/600x-concurrent_comp_unit.png

Jeremy: it's rather unfortunate timing.  six to twelve months ago, we
could have had this debate and it would have been fun, informative,
useful and productive.  it would also have been possible to bring you
in to the discussions that took place (Nov 2018 to around Apr-May
2019) between myself, Samuel Falvo, Mitch Alsup and the team either
here on this list, or on comp.arch.

however with the upcoming deadline, this kind of otherwise really
valuable discussion is, sadly, something that we *do not have time
for*.

retrospective recommendation that we reconsider and use Tomasulo would
not only result in us completely missing the upcoming Oct 2020
deadline, we would end up with an exceptionally poorly-performing,
limited and seriously problematic processor as well.  worse than that,
sadly i have to say that i am concerned that any retrospective
discussion would itself (aside from valuable corrections and
assistance in creating a comparative analysis) become a serious
distraction.

it's liitterallly taken 18 MONTHs to get to the point where we are now
with the code and the design.  we need to finish the design in a very
short timeframe (14 weeks), and as Yehowshua points out, due to the
complexity of this particular area of Computer Science, that in turn
implies that you'll need to trust me on the architectural analysis
(which took 5 months to complete, under Mitch's guidance).

so, my apologies for needing to be firm on this: the opportunity to
help exists within the time-frame and financial constraints that were
determined and set, over 18 months ago, before you joined.

therefore, if you'd like to help, then asking questions which are
related to *understanding* of the precise-augmented Scoreboard system,
targetted very specifically at allowing you to engage and assist us in
meeting that deadline, are very welcome.

anything other than that very specific goal-orientated focus, i am
really sorry, we really need to move it to one side, and for that, i
recommend comp.arch.

therefore, if you would like to discuss the merits of Tomasulo vs
Scoreboard (particularly those that pursue recommending Tomasulo
*over* Scoreboard) further, i recommend doing so on comp.arch, which
is where Mitch Alsup most regularly may be found.  with the right
interesting question, both he and many others will pitch in.

the one thing that you need to be aware of is that Mitch is both
hyper-intelligent, and second that he does not tolerate people who
cannot keep up with him, especially if they try to convince him of
something without having first demonstrated that they've understood
*him*.  he's well over 70, he's retired, he made enough "xxxx-you"
money on selling GPU patents to Samsung, so he doesn't have to put up
with anything that he doesn't want to :)

l.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Jeremy Singher]

I have finally finished parsing Mitch's chapters, and now I see what
the primary confusion is from. The scoreboard being developed here has
been referred to in the mailing lists as a "CDC 6600-like" scoreboard,
while in reality, the final design is far extended far beyond the
original CDC 6600 implementation. I would suggest not calling it a
"CDC 6600-like scoreboard" in the future, instead a "extended
scoreboard" is more accurate.

Let me try to compose my thoughts.
In order for OOO execution to be possible, write data from
speculatively executed instructions must be buffered, for in the case
of misspeculation, they must not be allowed to commit into the
architectural register file.

In Tomasulo's, speculative write data are held in physically renamed registers.
In the extended scoreboard scheme, speculative write data are held in
compute unit latches.
In Tomasulo's, branch dependency masks are used to drain misspeculated
instructions.
In extended scoreboard, the misspeculated instructions drain
themselves from the system.
Both schemes can be designed to achieve similar OOO behaviors.
Thus, one cannot draw a performance conclusion just by seeing which
scheme is being used.

Ideally the above text should be uncontroversial, please let me know
if there is a large misunderstanding here.

My opinion is that Tomasulo's is a more elegant and well-proven scheme
for achieving OOO execution, compared to an extended scoreboard. I
also argue that the extended scoreboard scheme does not present
significant power/area advantages compared to Tomasulo's. Given that
estimates on power/area tend to be wildly inaccurate, I think we will
have to wait for power analysis of LibreSoC to get a definitive answer
to this debate :).

I deeply apologize if any of my thoughts could be interpreted as
criticism. I fully support this kind of exploration in OOO design. I
also apologize if I seem to suggest a change in design, as I
understand that hardware design is not nearly as flexible as software
design.



On Sat, May 16, 2020 at 2:38 PM Luke Kenneth Casson Leighton
<lkcl@lkcl.net> wrote:
>
> On Sat, May 16, 2020 at 9:16 PM Jeremy Singher <thejsingher@gmail.com> wrote:
>
> > > However, the scoreboard is limited in that it does not handle WAR and WAW
> > > hazards very well.
> >
> > > the original 6600 handles WAR extremely well, only stalling on WaW
> > > condition, detection which did not matter greatly because the pipelines
> > > were only at most 2 stages long anyway (Mitch only noticed after rereading
> > > last year that the FP ADD of the 6600 was 2 stage pipelined.  no academic
> > > literature has acknowledged or noticed this).
> >
> > I would argue that the CDC6600 handles WAR better than an in-order
> > machine, but not better than a machine with full renaming.
>
> Jeremy: 6600 designs *do* have register "renaming".  i said already,
> a number of times.  and if you have studied and listened to the Academic
> literature - and believe it - you are misinformed by that literature, and will
> continue to be misinformed by that literature.
>
> please understand: i have spent over nine months studying this, learning
> from Mitch Alsup, one of the world's leading high-performance computing
> architects, implementing this and it's not going to get chucked out to
> implement something that takes six MONTHS to fully understand is a
> better design, to be replaced by something that would take another six
> months to implement *and we then could not do multi-issue on top of it*.
>
> please therefore take the time to understand this design, particularly when
> we are right in the middle of a time-critical deadline.
>
> > In the
> > CDC6600, the scoreboard would block the completion of the W inst until
> > the R inst has read its operands. In Tomasulo, the W inst can proceed
> > immediately before the R inst is ready, since the destination register
> > is physically renamed.
>
> Jeremy: the "renaming" carried out by the 6600 does exactly this same
> job.
>
> Academics unfortunately have *only* studied the *patent* - not Thornton's
> book nor read Mitch Alsup's book chapters which he wrote based on 30+
> years experience of implementing 6600-style execution in the 88100,
> AMD K9 and AMD Opteron Series of CPUs.
>
> the 6600 patent *ONLY* covers the Q-Tables.
>
> the description of the Q-Tables does *NOT* include the Computation Units
> where the operand latches are.
>
> these latches *ARE* the "renamed registers" (actually, nameless registers).
>
> > >  ROB is a CAM,
> >
> > What? This is not true. Instructions in flight track their index in
> > the ROB, at completion, they write success bit according to the index
> > they have. Not a CAM. Simple addressed write.
>
> ah appreciated the correction.  i knew there were CAMs somewhere.
> when i discovered that, i dropped further investigation.
>
> i am however aware that if you wish to do multi-issue, the ROB will need
> to be multi-ported SRAM.  without multi-porting the SRAM, you simply
> cannot write multiple entries.
>
> this is where the problems (read: nightmares) for Tomasulo start to manifest,
> and progress from there through multiple Common Data Bus channels into
> multi-ported (NxR, NxW) CAMs at the Reservation Station level.
>
> i've been through this, ok, over 18 months ago.
>
> by contrast, when converted to single-bit unary vectors, each entry in the
> Matrices (covered by a single SR NAND latch) is *independent*.  multiple
> of those may be set in a single cycle.
>
> there has been a *massive* amount of thought gone into this, ok?
>
>
> > > given that Intel processors use Tomasulo, we start to see why Intel
> > > processors suck so much power.
> >
> > Power consumption on a processor is primarily from high switching
> > rates on large register files, and SRAM leakage.
>
> given that we're planning around 28 Function Units and have a 128-entry
> register file, the calculations that i did for that would put the Dependency
> Matrices around a whopping 50,000 gates, using SR NAND Latches.
>
> (if we did it as DFFs it would be an unacceptable 250,000 gates)
>
> > The scheduling logic is a miniscule portion.
>
> not for the large-scale out-of-order systems that we are planning to
> do, it's not,
> due to the large sized GPU register file (128 entries.  actually 2x
> 128 - 128 FP,
> 128 INT).
>
> we may have to *down*-scale via a regfile cache to cope with that.
> discussion for another time.
>
>
> > Intel cores are power-hungry because
> > historically they have targeted the desktop/workstation market, and do
> > not have as many decades of experience as other firms optimizing for
> > power-constrained environments.
>
> appreciated the insight.
>
> > Also almost all modern OOO cores use Tomasulo because Tomasulo is
> > better at avoiding false hazards compared to scoreboarding.
>
> in the versions of scoreboards implemented by academics - because they
> fail to comprehend them properly - because they only read the patent
> on Q-Tables - yes.
>
> in the version that Mitch Alsup - a world leading expert on Scoreboard
> Mechanics - taught me:
>
> no.
>
> also, the IIT Madras RISE Group Shakti E-Class processor is *also* a
> scoreboard design.  this was designed "from first principles" by Professor
> Kamatoti, and it was only through learning from Mitch that i was able
> to recognise what he had done as being identical to a 6600 scoreboard.
>
> > Tomasulo
> > also makes it easier to implement precise traps, something which is
> > notoriously challenging  with just a scoreboard.
>
> no it is not.  this is *yet again* complete factual misinformation and ignorance
> from Academic literature.
>
> can i ask: have you read Patterson's book?  this is the usual place where
> most people get their misinformation from.  the release of this misinformation
> was why Mitch Alsup - a very pissed-off and exasperated Mitch - wrote those
> two chapters, back around... 2008 (i think).
>
> precise exceptions, traps, interrupts, branch speculation and predication may
> all be achieved through "shadowing".  see the latter part of the 2nd chapter of
> Mitch's book.
>
> see the section "Shadowing" in this page:
> https://libre-soc.org/3d_gpu/architecture/6600scoreboard/
>
> it's very simple: you hold write until such time as the opportunity for "damage"
> is cleared.  Operand Forwarding allows instructions to run ahead, using results
> that would otherwise have to be received only first by going through
> the Regfile.
>
> *this is already implemented* and the unit tests demonstrated it as
> *fully functional*
>
>
> > Lets be clear here. The CDC6600 AVOIDS WAR and WAW hazards by BLOCKING
> > younger instructions (See pg3 in Mitch Alsup's text). Tomasulo's
> > supports OOO execution PAST WAR and WAW hazards, enabling the core to
> > exploit more ILP across its functional units.
>
> this is only because the 1965 CDC6600 did not have an Operand Forwarding Bus
> (instead it used the regfile as a same-clock forwarding bus, by reading on one
> edge and writing on the other.  this unfortunately meant that without an OpFwd
> Bus, stall had to occur).
>
> this DOES NOT INHERENTLY MEAN that **ALL** Scoreboards are rubbish.
>
> if you add an Operand Forwarding Bus (equivalent to the CDB in Tomasulo)
> the exact same parity functionality is achieved (without the severe downsides).
>
>
> > Luke, I agree with you that the reservation station rows in Tomasulo's
> > can be equivalent to the operand latches in Scoreboarding. However,
> > adding many operand-latches (by having many functional units) does not
> > scale as nicely as just adding a few more rows to Tomasulo's
> > reservation stations.
>
> this is incorrect.  you may be under the mistaken impression that the pipelines
> duplicated as well.  this clearly would be inadviseable because the FU's job
> is to serve one and only one result.  therefore you know that this is a mistaken
> understanding, and that consequently you have multiple FUs for the same
> pipeline.  (termed "Concurrent Computation Unit").
>
> therefore it's minimal.  excluding the latches themselves (the rows
> have latches, the
> CompUnits - aka FunctionUnits have latches, the total number of latches after
> flattening is the same), the number of gates added by each CompUnit is
> minimal: around... 80, maximum.
>
> that's peanuts.
>
> > Especially since designing the CAMs for
> > Tomasulo's is a well studied, and well-understood problem
>
> the key is the combination of those two:
>
> A) well-understood... *and*
> B) a problem.
>
> have a look at how multi-issue is done using Tomasulo.  it's shockingly bad.
>
>
> > > 3. Expand the number of RSes so that if you were to count the total number
> > > of places operands are stored, they are the same.
> >
> > > (another way to put this is, "flatten all 2D RSes into 1D")
> >
> > This does not scale well. A 2D CAM is far more area efficient than a
> > 1D flattened vector.
>
> a 1D bitvector.  a single, 1D bitvector.  of 32 bits in length.
>
> compared to 32 5-bit binary addresses.
>
> just the latches alone, that math does not stack up.
>
> now let's take it to 2D.
>
> let's say that we have 10 FUs and 32  Regs.
>
> in Tomasulo that is a 5-bit address.  10x5 = 50 DFF latches.  DFF
> latches are 10 gates so that's 500 gates.
>
> in 6600 that is 10x32 = 320 SR NAND Latches.  SR NAND latches are 2
> gates.  320x2 is 640 gates.
>
> not very different in terms of latch count, is it?
>
>
> now let's do the address-checking.
>
> to check a register as "active" - bear in mind that this is per
> register and consequently
> if doing multi-issue that's a batch of address-checking gates *per register*.
>
> for Tomasulu, this requires 10x5 = 500 XOR gates (plus some
> multi-input AND gates on each row)   XOR is 4 gates, so that's 2,000
> gates *PER MULTI-ISSUE LANE*.
>
> for the 6600, that's 10x32 **AND** gates.  320 AND gates.
>
> and of those 320 AND gates, they are re-useable for multi-issue
> WITHOUT REQUIRING ADDITIONAL GATES.
>
> so that is 320 AND gates for single-issue
> that is zero extra gates for dual-issue
> that is zero extra gates for quad-issue
> zero extra gates for octal-issue.
>
> start to make sense now, why unary encoding as 1D vectors is less
> gates and less power than binary CAMs?
>
> if you want to check multiple unary bits (a critical prerequisite for
> multi-issue), all you do is... set... multiple... bits.
>
> what happens?  multiple AND gates fire.  in the same clock.   and
> that's exactly what we want.
>
>
> > > 7. rename RSes to "Function Units" (actually in Thornton's book the phrase
> > > "Computation Units" is used)
> >
> > So a Tomasulo's design with 4 rows now becomes a "flattened" design
> > with 4 functional units? The additional cost of the functional units
> > is enormous.
>
> no, it's 4 x 80 gates (approx).  that's not 4x duplication of ALU
> pipelines *behind* those 4 Function Units.  see Mitch's 2nd Chapter,
> "Concurrent Computation Unit"
>
> https://libre-soc.org/3d_gpu/architecture/6600scoreboard/600x-concurrent_comp_unit.png
>
> Jeremy: it's rather unfortunate timing.  six to twelve months ago, we
> could have had this debate and it would have been fun, informative,
> useful and productive.  it would also have been possible to bring you
> in to the discussions that took place (Nov 2018 to around Apr-May
> 2019) between myself, Samuel Falvo, Mitch Alsup and the team either
> here on this list, or on comp.arch.
>
> however with the upcoming deadline, this kind of otherwise really
> valuable discussion is, sadly, something that we *do not have time
> for*.
>
> retrospective recommendation that we reconsider and use Tomasulo would
> not only result in us completely missing the upcoming Oct 2020
> deadline, we would end up with an exceptionally poorly-performing,
> limited and seriously problematic processor as well.  worse than that,
> sadly i have to say that i am concerned that any retrospective
> discussion would itself (aside from valuable corrections and
> assistance in creating a comparative analysis) become a serious
> distraction.
>
> it's liitterallly taken 18 MONTHs to get to the point where we are now
> with the code and the design.  we need to finish the design in a very
> short timeframe (14 weeks), and as Yehowshua points out, due to the
> complexity of this particular area of Computer Science, that in turn
> implies that you'll need to trust me on the architectural analysis
> (which took 5 months to complete, under Mitch's guidance).
>
> so, my apologies for needing to be firm on this: the opportunity to
> help exists within the time-frame and financial constraints that were
> determined and set, over 18 months ago, before you joined.
>
> therefore, if you'd like to help, then asking questions which are
> related to *understanding* of the precise-augmented Scoreboard system,
> targetted very specifically at allowing you to engage and assist us in
> meeting that deadline, are very welcome.
>
> anything other than that very specific goal-orientated focus, i am
> really sorry, we really need to move it to one side, and for that, i
> recommend comp.arch.
>
> therefore, if you would like to discuss the merits of Tomasulo vs
> Scoreboard (particularly those that pursue recommending Tomasulo
> *over* Scoreboard) further, i recommend doing so on comp.arch, which
> is where Mitch Alsup most regularly may be found.  with the right
> interesting question, both he and many others will pitch in.
>
> the one thing that you need to be aware of is that Mitch is both
> hyper-intelligent, and second that he does not tolerate people who
> cannot keep up with him, especially if they try to convince him of
> something without having first demonstrated that they've understood
> *him*.  he's well over 70, he's retired, he made enough "xxxx-you"
> money on selling GPU patents to Samsung, so he doesn't have to put up
> with anything that he doesn't want to :)
>
> l.
>
> _______________________________________________
> libre-riscv-dev mailing list
> libre-riscv-dev@lists.libre-riscv.org
> http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Yehowshua]

> On May 16, 2020, at 6:45 PM, Jeremy Singher <thejsingher@gmail.com> wrote:
> 
> Let me try to compose my thoughts.
> In order for OOO execution to be possible, write data from
> speculatively executed instructions must be buffered, for in the case
> of misspeculation, they must not be allowed to commit into the
> architectural register file.

This sounds correct to me.

> I think we will
> have to wait for power analysis of LibreSoC to get a definitive answer
> to this debate :).

Probably.

Yehowshua
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Scoreboard vs Tomasulo

[Luke Kenneth Casson Leighton]

On Sat, May 16, 2020 at 11:45 PM Jeremy Singher <thejsingher@gmail.com> wrote:
>
> I have finally finished parsing Mitch's chapters, and now I see what
> the primary confusion is from. The scoreboard being developed here has
> been referred to in the mailing lists as a "CDC 6600-like" scoreboard,
> while in reality, the final design is far extended far beyond the
> original CDC 6600 implementation.

yes.  well...  far beyond it in *functionality*, but not that far in
terms of gate
count [caveat: if we were only doing the same 3 regfiles, A B and X.
as we are doing 32x INT, 32x FP, plus SPR regs *and* Condition Regs
all under the control of Dependency Matrices, a direct comparison is unfair]

oh.  ah.  one definite enhancement: in the diagrams you see in Mitch's
chapters, there is only one GO-READ and only one GO-WRITE.  one GO-READ
indicates a desire to request *multiple* Regfile Read ports.

because POWER9 has in some cases up to *five* incoming registers and
*three* outgoing registers (in different Regfiles - INT/FP, SPR, CR), we needed
to split those out into individual lines.  GO-READ1/READ-REQ1,
GO-READ2/READ-REQ2 etc.

currently i am struggling with this FSM which implements 3-operand in,
immediates, 2-operand out *and* manages the address, GO-ST, address
exceptions and deals with POWER9 LD/ST "Update" mode.

https://libre-soc.org/3d_gpu/ld_st_comp_unit.jpg

this i reaaaally need some help with.


> I would suggest not calling it a
> "CDC 6600-like scoreboard" in the future, instead a "extended
> scoreboard" is more accurate.

appreciated.  if you could consider helping change the wiki to reflect this (and
help us to maintain the advantages below) this would be fantastic.

i've used a range of terms - "augmented 6600 scoreboard", and others.
the historic
link and the effective revival of something long-forgotten which was
both elegant
and staggeringly gate-efficient (out of necessity): i don't want to use the word
"political" in any sentence however there is a considerable publicity
"wow factor"
to doing something as surprising as reviving a 55-year-old design.

since interacting with Mitch Alsup last year, you're the first additional person
to actually do any kind of significant analysis: consequently terminology has
not really been discussed... because there wasn't anyone to discuss it with!

> Let me try to compose my thoughts.
> In order for OOO execution to be possible, write data from
> speculatively executed instructions must be buffered, for in the case
> of misspeculation, they must not be allowed to commit into the
> architectural register file.

mis-speculation, LD/ST page-faults, predicated instructions where the operand
that contains the bitmask is in a register that has not been read yet, and even
simple external NMI interrupts: all of these need a means and method of
killing in-flight data, yes.

> In Tomasulo's, speculative write data are held in physically renamed registers.
> In the extended scoreboard scheme, speculative write data are held in
> compute unit latches.

these two effectively having a very low Levenshtein distance, but not quite
zero.

* in Tomasulo, the reason for the explicit renaming is down to the 2D
nature of RS's.
  hence the ROB row number needs to be stored in the RS

* in extended-scoreboard, the FU's index / column-number *is* that "name";
  with there only *being* a one-to-one path, there is no need for a
name *at all*,
  hence why i refer to the CU latches as "nameless registers".

  the ROB row number *is* the FU number which *is* the renamed-register-number
  and consequently by being one-to-one in every aspect, needs no name.

this aspect of 6600 scoreboards - the complete lack of naming and even the
lack of significance drawn to it even by James Thornton himself in his book,
and certainly in the patents, is why the 6600 is *believed* not to have register
renaming of *any* kind.

ah - one other really critical thing (i will add to the page):

* Tomasulo ROB Row numbers preserve instruction order through sequential
   numbering by the actual row number

* 6600 scoreboards preserve instruction order in the FU-FU Matrix with
  a NxN bit-level matrix of SR Latches implementing a "linked list" (the
  Directed Acyclic Graph) of ordered dependencies, using the *register*
  dependencies at the time the instruction is inserted into the *FU-REGS*
  Matrix to determine the FU-FU DAG.  this has more in common with a
  "linked list" data structure, to use software terminology.

this latter aspect is a crucial one that even James Thornton notes is really
*really* complex to explain, yet paradoxically requires an extraordinarily
small number of actual gates to implement.

however understanding the difference between the (binary) ROB Row
numbering (which preserves the DAG of instruction order by way
of binary-addressed instruction ROB Row N+1 being a greater index
than binary-addressed instruction ROB Row N) and this
"functionally-equivalent" linked-list system equally successfully preserving
a DAG but using unary SR Latches to do so, is key to recognising
and appreciating that Tomasulo and Scoreboards *really are* functionally
directly equivalent.

here it should become clear as to another aspect of why multi-issue is
problematic in Tomasulo.  where detection of "readiness to commit" in
a single-issue Tomasulo is a simple matter of checking *one* row
"is your result ready, if yes commit it and cyclically increment the pointer
to the ROB row to the next item, ready for test-or-commit on the *next*
cycle", this must be updated to testing of *multiple* row "readiness"
states, which clearly requires a multi-ported SRAM.

this is where "solutions" that i have seen actually stratify the ROB
into 4 (or N) interleaved banks.  crossbars are required... it's where
the nightmares begin.


> In Tomasulo's, branch dependency masks are used to drain misspeculated
> instructions.

are these unary-encoded or binary-encoded?  by masks i would expect them
to be unary-encoded and thus allow multiple bits to be set, allowing multiple
operations to be cancelled.

if not, and binary masks are used, then each unit must have an arithmetic
comparator, "is my binary index less than or equal to the branch dependency
mask, if less than, then this means i must die".

the Shakti 6600-like (extended) scoreboard system in their E-Class design
uses *binary* numbering, which means that they use that scheme.  effectively
the mask is not a mask per se, it is a shortened version of the instruction
sequential number.

BOOM uses unary bit-masks.  i think.  i had a brief conversation about it
with someone, somewhere, a few months back.

i greatly prefer the unary system.  it allows interleaving of multiple
simultaneous
cancellations to *automatically* sort themselves out, by ORing the cancellations
together to produce the mask!

this is an "accidental" side-effect of the Shadow Matrix concept,
described below.

a binary-indexed cancellation system requires a *central resource* to calculate
the lowest-numbered "index", followed by broadcasting that index across the
one (and only one) permitted cancellation datapath.


> In extended scoreboard, the misspeculated instructions drain
> themselves from the system.

ah no.  we'll be cancelling them explicitly, via an unary bit-mask,
that is global
and combinatorial in nature, extending right the way through the entirety
of the Function Unit, CompUnits *and* the associated pipelines.

the unary mask is basically a conversion of the *binary* indices of the
CompUnit number on any given Concurrent Computation Unit.  0-3
in this diagram here:
https://libre-soc.org/3d_gpu/architecture/6600scoreboard/600x-concurrent_comp_unit.png

thus in that case the unary cancellation mask would be 4-bit wide.

these are linked directly to - and are equal to - the "GO_DIE" signals in *this*
diagram:
https://libre-soc.org/3d_gpu/shadow.jpg

note in that diagram the existence per cell in the Dependency Matrices of *two*
Shadow Latches.

the first Shadow Latch is for the first Branch Unit to indicate "speculative
execution underway as controlled and directed by Branch Unit 1".

the second Shadow Latch is for the *second* Branch Unit to do likewise.

a third Shadow Latch exists for predicated execution.  one bit each of the
predicate register (once read) will be thrown at the "FAILED-3" line if that
bit is zero, and at the "SUCCESS-3" line if that bit is 1.  if "failed" is
asserted, then so is the "GO_DIE".  that "GO_DIE", being connected to
the cancellation masks, *INSTANTLY* kills not just the register hazard
allocation, it resets the Function Unit to idle *and* kills the partial result
generation actually in the pipeline itself.

if "success" is asserted, the shadow latches are cleared, and "WRITE_REQUEST"
is no longer prevented and prohibited from asserting.  the result (if it was
ready), is allowed to proceed "I now request a Regfile Write Port".
if the result
happened not to be ready, then when the result *is* ready it will not
be held up.

a fourth Shadow latch exists for cancellation of operations in the "shadow"
of a LD/ST Unit's page-fault / address exception.

as you can see this requires a trivial number of gates per cell, however
it is *each* exception or cancellation opportunity that  requires these Shadows,
on a *per FU basis*, across all *other* FUs.

consequently i designed a "Shadow Matrix" that takes care of that.

> Both schemes can be designed to achieve similar OOO behaviors.
> Thus, one cannot draw a performance conclusion just by seeing which
> scheme is being used.

no.  hence why we haven't done that kind of analysis: we're focussing
on implementation.

> Ideally the above text should be uncontroversial, please let me know
> if there is a large misunderstanding here.

nono looks great, really appreciate the insights raised.  the missing piece
is the true significance of the role of the Shadow Matrices as being *the*
provision of precise exception capability, speculation, predication, everything.

yes i am aware that without "predication prediction" yes those are two
different words we end up with exactly the same situation as for Branch
Prediction: predicated operations that are issued, cause FUs to become
full / allocated, that are cancelled moments later because the majority
of the predicate register turns out to be full of zeros.

this is *not* a high priority item right now :)

> My opinion is that Tomasulo's is a more elegant and well-proven scheme
> for achieving OOO execution, compared to an extended scoreboard. I
> also argue that the extended scoreboard scheme does not present
> significant power/area advantages compared to Tomasulo's.

Mitch Alsup is better equipped, from 40 years experience at gate-level
design (which he kept and *refined* when working for AMD, hence some
of the blistering withering words to less experienced, less-knowledgeable
people trying to tell him that HDLs were inherently better), to answer this
one.

> Given that
> estimates on power/area tend to be wildly inaccurate, I think we will
> have to wait for power analysis of LibreSoC to get a definitive answer
> to this debate :).

if we have to use DFFs rather than SR NAND, the argument is instantly
won by Tomasulo because we have quite literally a 5-fold increase in
gate count right across the board, in every single Matrix.  Shadow,
FU-Regs, FU-FU - the lot.  everywhere you see an SR-Latch diagram
(2 gates), instead add 10 gates for a DFF.

for the 180nm ASIC this would jump approximately.... mmm... 10,000 gates
up to 50,000.

we may have to do that.  it will suck.

> I deeply apologize if any of my thoughts could be interpreted as
> criticism. I fully support this kind of exploration in OOO design. I
> also apologize if I seem to suggest a change in design, as I
> understand that hardware design is not nearly as flexible as software
> design.

appreciated, Jeremy.  if this discussion was taking place 6 to 12 months ago
the "mode" we would be in was radically different.  now is "full-bore
time-critical implementation mode", and will be until the other side
of Oct 2020.

l.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

[libre-riscv-dev] LD/ST Comp Unit FSM (was: Re: Scoreboard vs Tomasulo)

[Cesar Strauss]

Greetings,

On 05/16/2020 21:53, Luke Kenneth Casson Leighton wrote:
> because POWER9 has in some cases up to *five* incoming registers and
> *three* outgoing registers (in different Regfiles - INT/FP, SPR, CR), we needed
> to split those out into individual lines.  GO-READ1/READ-REQ1,
> GO-READ2/READ-REQ2 etc.
> 
> currently i am struggling with this FSM which implements 3-operand in,
> immediates, 2-operand out *and* manages the address, GO-ST, address
> exceptions and deals with POWER9 LD/ST "Update" mode.
> 
> https://libre-soc.org/3d_gpu/ld_st_comp_unit.jpg
> 
> this i reaaaally need some help with.

I'm willing to lend a hand in this, and/or any other tasks the group
deem appropriate.

Do you have already have any code, unit test and/or testbench for this
module? A timing diagram would also be welcome.

Regarding myself, I do have experience in digital circuit design, and
finite state machines in particular, which I mostly apply to FPGA
programming.

My field of work is in data acquisition and control, for scientific
instruments, at the astrophysics department of a research institute (but
I am an engineer, not an astronomer). I apply microcontrollers and FPGAs
at several research projects. With the rise of single-board computers,
I'm starting to use them as well.

My main attraction to this project is having free firmware and boot
loaders, and also full documentation, freely available and complete, by
principle.

I have read the Charter at https://libre-soc.org/charter/, and agree
with it.

I can use only my free time (weekends and evenings).

Regards,

Cesar

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] LD/ST Comp Unit FSM (was: Re: Scoreboard vs Tomasulo)

[Luke Kenneth Casson Leighton]

On Sun, May 17, 2020 at 6:25 AM Cesar Strauss <cestrauss@gmail.com> wrote:

> > https://libre-soc.org/3d_gpu/ld_st_comp_unit.jpg
> >
> > this i reaaaally need some help with.
>
> I'm willing to lend a hand in this, and/or any other tasks the group
> deem appropriate.

that's really appreciated, Cesar.

> Do you have already have any code, unit test and/or testbench for this
> module? A timing diagram would also be welcome.

yes, it's here:
https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/experiment/compldst_multi.py;hb=HEAD

the timing diagram is auto-generated when running that code (as its
own unit test: python3 experiment/compldst_multi.py).  it can be
loaded with "gtkwave".  to save you some time adding 100+ signals to
gtkwave i'm attaching a pre-saved diagram, i even set some pretty
colours in it :)

also a second one for the L0 cache/buffer to which CompLDSTUnit is
connected.  this is run as "python3 experiment/l0_cache.py"

i will raise a bugreport about it as this should be discussed there.

> Regarding myself, I do have experience in digital circuit design, and
> finite state machines in particular, which I mostly apply to FPGA
> programming.
>
> My field of work is in data acquisition and control, for scientific
> instruments, at the astrophysics department of a research institute (but
> I am an engineer, not an astronomer). I apply microcontrollers and FPGAs
> at several research projects. With the rise of single-board computers,
> I'm starting to use them as well.
>
> My main attraction to this project is having free firmware and boot
> loaders, and also full documentation, freely available and complete, by
> principle.

that's really appreciated.

> I have read the Charter at https://libre-soc.org/charter/, and agree
> with it.

thank you Cesar.
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] LD/ST Comp Unit FSM (was: Re: Scoreboard vs Tomasulo)

[Cesar Strauss]

Em 05/17/2020 07:07, Luke Kenneth Casson Leighton wrote:
> On Sun, May 17, 2020 at 6:25 AM Cesar Strauss <cestrauss@gmail.com> wrote:
> 
>>> https://libre-soc.org/3d_gpu/ld_st_comp_unit.jpg
>>>
>>> this i reaaaally need some help with.
>>
>> I'm willing to lend a hand in this, and/or any other tasks the group
>> deem appropriate.
> 
> that's really appreciated, Cesar.

I'm glad I can help.

> 
>> Do you have already have any code, unit test and/or testbench for this
>> module? A timing diagram would also be welcome.
> 
> yes, it's here:
> https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/experiment/compldst_multi.py;hb=HEAD
> 
> the timing diagram is auto-generated when running that code (as its
> own unit test: python3 experiment/compldst_multi.py).  it can be
> loaded with "gtkwave".  to save you some time adding 100+ signals to
> gtkwave i'm attaching a pre-saved diagram, i even set some pretty
> colours in it :)
> 
> also a second one for the L0 cache/buffer to which CompLDSTUnit is
> connected.  this is run as "python3 experiment/l0_cache.py"
> 
> i will raise a bugreport about it as this should be discussed there.

I saw it (https://bugs.libre-soc.org/show_bug.cgi?id=318). I'll assign
it to myself then, if you don't mind?

I think I have enough information available, to proceed. I'm beginning
already to have some understanding of the problem at hand.  I'll ask if
I need anything. I'll give feedback as I go on.

I've been following the mailing list for some time, with interest, but
skimming over more technical details. I'll try to become more up to speed.

I did attend an introductory course on Computer Architecture, so I do
have a little background on the subject.

Regards,

Cesar

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] LD/ST Comp Unit FSM (was: Re: Scoreboard vs Tomasulo)

[Luke Kenneth Casson Leighton]

On Sun, May 17, 2020 at 1:59 PM Cesar Strauss <cestrauss@gmail.com> wrote:
>
> Em 05/17/2020 07:07, Luke Kenneth Casson Leighton wrote:
> > On Sun, May 17, 2020 at 6:25 AM Cesar Strauss <cestrauss@gmail.com> wrote:

> I saw it (https://bugs.libre-soc.org/show_bug.cgi?id=318). I'll assign
> it to myself then, if you don't mind?

yes please do.  ahh i *might* have "fixed" it - however this is such
complex code i cannot be truly certain.

transferring to the bugreport.

l.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] LD/ST Comp Unit FSM (was: Re: Scoreboard vs Tomasulo)

[Luke Kenneth Casson Leighton]

btw Cesar you've been tracking the list for a while, so you know of this:
https://libre-soc.org/HDL_workflow

feel free to send me an ssh public key.

l.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] LD/ST Comp Unit FSM (was: Re: Scoreboard vs Tomasulo)

[Cesar Strauss]

On 05/17/2020 19:23, Luke Kenneth Casson Leighton wrote:
> btw Cesar you've been tracking the list for a while, so you know of this:
> https://libre-soc.org/HDL_workflow

Indeed.

> feel free to send me an ssh public key.

Gladly. It is attached.

Regards,

Cesar
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] LD/ST Comp Unit FSM (was: Re: Scoreboard vs Tomasulo)

[Luke Kenneth Casson Leighton]

you may have to send that privately, the list only allows certain attachments.

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Sun, May 17, 2020 at 11:42 PM Cesar Strauss <cestrauss@gmail.com> wrote:
>
> On 05/17/2020 19:23, Luke Kenneth Casson Leighton wrote:
> > btw Cesar you've been tracking the list for a while, so you know of this:
> > https://libre-soc.org/HDL_workflow
>
> Indeed.
>
> > feel free to send me an ssh public key.
>
> Gladly. It is attached.
>
> Regards,
>
> Cesar
> _______________________________________________
> libre-riscv-dev mailing list
> libre-riscv-dev@lists.libre-riscv.org
> http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev