[libre-riscv-dev] Fall 2022 Interfaces

[libre-riscv-dev] Fall 2022 Interfaces

[Yehowshua]

Was talking with Pine64 and Raptor about Fall2022 interfaces which I’m trying to finalize.
We can do everything including HDMI, ethernet, DDR4(via OpenCAPI OMI), and PCIE with SERDES lanes.

We need 10rx and 14tx SERDES.
4rx and 4tx for the OMI will need to run at 25GHz for DDR4 over OMI.
@Luke, is this possible for 45nm? Seems like it should be fine.

What type of expenses are we talking about for having Symbiotic EDA design the SERDES?

We’re going to need at least four different clock speeds - we will have to think about how to do this PLL wise - and down clock where possible.

https://libre-soc.org/45nm_Fall2022/?updated <https://libre-soc.org/45nm_Fall2022/?updated>

Yehowshua
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Yehowshua]

> Was talking with Pine64 and Raptor about Fall2022 interfaces which I’m trying to finalize.

Actually, wouldn’t say finalize so much as get a pretty good idea.
Things will obviously become more set in stone as we approach Fall 2022.

Yehowshua


_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Jacob Lifshay]

On Thu, Jun 11, 2020, 19:59 Yehowshua <yimmanuel3@gatech.edu> wrote:

> Was talking with Pine64 and Raptor about Fall2022 interfaces which I’m
> trying to finalize.
> We can do everything including HDMI, ethernet, DDR4(via OpenCAPI OMI), and
> PCIE with SERDES lanes.
>

hdmi has issues with libre licensing and DRM content, if we're going that
fast, we should just go with display port or similar, since it's an open
standard, I think.


> We need 10rx and 14tx SERDES.
> 4rx and 4tx for the OMI will need to run at 25GHz for DDR4 over OMI.
>

25GHz seems *really* fast, probably impractically so unless going to *lots*
of effort, did you mean 2.5GHz?

Jacob
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Cole Poirier]

> On Jun 11, 2020, at 9:01 PM, Jacob Lifshay <programmerjake@gmail.com> wrote:
> 
> On Thu, Jun 11, 2020, 19:59 Yehowshua <yimmanuel3@gatech.edu> wrote:
> 
>> Was talking with Pine64 and Raptor about Fall2022 interfaces which I’m
>> trying to finalize.
>> We need 10rx and 14tx SERDES.
>> 4rx and 4tx for the OMI will need to run at 25GHz for DDR4 over OMI.
>> 
> 
> 25GHz seems *really* fast, probably impractically so unless going to *lots*
> of effort, did you mean 2.5GHz?

Maybe 25GB/s? “The OMI DDR4 Differential DIMM memory module supports a data throughput rate of 25.6GB/s with a latency of 40ns and densities up to 256GB. The new 84-pin DDR4 DDIMM is intended for use in standard server environments, using a serial interface and a differential Data Buffer (dDB) from Microchip. The DDIMM supports IBM’s P9 AIO and P10 processors’ memory attached architecture. The P9 AIO and P10 memory bus is defined with one read port and one write port per channel, each having eight unidirectional differential lanes supporting 25.6 Gbps data rate over the OMI direct attached to the DDIMM.”
https://www.eenewseurope.com/news/ddr4-differential-dimm-grows-opencapi-ecosystem

Still that’s for power 10 and late generation power 9 server processors.

Cole
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Yehowshua]

> 
> hdmi has issues with libre licensing and DRM content, if we're going that
> fast, we should just go with display port or similar, since it's an open
> standard, I think.

Perhaps - was reading Dave Shah’s convo on gitter…

He said Display Port is an absolute pain…

Yehowshua
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Yehowshua]

> On Jun 12, 2020, at 12:01 AM, Jacob Lifshay <programmerjake@gmail.com> wrote:
> 
> 25GHz seems *really* fast, probably impractically so unless going to *lots*
> of effort, did you mean 2.5GHz?

I think its actually 5GHz - currently checking with somebody from IBM.

Yehowshua
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Luke Kenneth Casson Leighton]

On Friday, June 12, 2020, Jacob Lifshay <programmerjake@gmail.com> wrote:

> On Thu, Jun 11, 2020, 19:59 Yehowshua <yimmanuel3@gatech.edu> wrote:
>
> > Was talking with Pine64 and Raptor about Fall2022 interfaces which I’m
> > trying to finalize.
> > We can do everything including HDMI, ethernet, DDR4(via OpenCAPI OMI),
> and
> > PCIE with SERDES lanes.
> >
>
> hdmi has issues with libre licensing and DRM content,


indeed.  DVI however does not (and is pin compatible).  HDMI is just a
protocol over DVI.


>  if we're going that
> fast, we should just go with display port or similar, since it's an open
> standard, I think.


i added Richard Herveille's RGBTTL to the list.  a TI TFP410a will convert
to at least 1600 x 1200 DVI no problem.

the TFP410a is a well known solution.

if we make it DDR RGBTTL then 1920x1080 is also no problem, via some
Chrontel ASICs.



>
>
> > We need 10rx and 14tx SERDES.


that is one hell of a lot.


> > 4rx and 4tx for the OMI will need to run at 25GHz for DDR4 over OMI.


that may not be enough to handle framebuffers as well as run an OS.
framebuffers are scarily demanding of memory bandwidth and they absolutely
MUST have total absolute top priority.

you cannot have a video scan line go blank.  evverrr.

to give some idea: 1920 x 1080 is 2 million pixels.  each pixel is 4
bytes.   that is 64 million bytes per frame.  60 frames? that is 360
MEGABYTES per second.

DDR3 800 Mhz is only 6400 MB per second so we are looking at 20 percent of
memory bandwidth taken by feeding the framebuffer!!!!

if you want 4k that is now 1200 MEGABYTES per second just to feed the
frickin framebuffer!

doesn't even matter if it contains blank lines, that's 1.2 gigabytes per
second full of zeros!




> >
>
> 25GHz seems *really* fast, probably impractically so unless going to *lots*
> of effort, did you mean 2.5GHz?


no it really is 25GHz.


we will need Rudi from ASICS.ws for this.  he can handle that kind of
work.  when i went to visit him he was doing some SERDES PHY related work
at these speeds.

i do not know if 45nm can handle it, though.

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Luke Kenneth Casson Leighton]

(offlist conversation moved onlist after removing names).

phone messing formatting.  blech.  sorry.

the bandwidth requirements for video framebuffer are so insane i am leaning
towards suggesting use a multi scanline compression hardware algorithm.
similar to the old fax machines, huffman encoding, blah blah.

multi scanline because this will pick up vertical compression opportunities.



On Friday, June 12, 2020, Yehowshua <yimmanuel3@gatech.edu> wrote:

> two uses
>
1. Stand alone SBC
2. GPU for POWER systems, and any system really.

yep.  this you can see on the original pages and descriptions i created.

glad to see you also worked it out.



> We can then literally provide identical GPU drivers to the host machine,
> except now, commands are sent over PCIE to LibreSOC where they are
> then issued as instructions on the hybrid CPU/GPU.
>

indeed.  with associated insanity and complexity, for which an entire
special team will be needed.


And remember, LibreSOC already has video output -
>

only if Richard Herveille's RGBTTL interface is included.

 so just send out the frame!
>

yyep :)

fortunately PCIe just memorymaps the framebuffer (i think)

so for nonaccelerated video it at least becomes braindead simple.

what is nice however is that by running an entire OS on the GPU, and it
being a full OS, a protocol can be invented which transfers video.

OR

just run the XServer *on the GPU*.

run xhost +

and... um... that's it.

done.

no need to write any drivers.

just treat the graphics card as if it was a networked Xserver.



> OK now to talk performance. When doing the discrete GPU, we can lift power
> restrictions.
> So lets say we double vector land widths, run the chip at 2GHz…
>
> We're talking 100 maybe 200GFLOPS.
>

yes.

it comes down to memory bandwidth at L0CacheBuffer.

4 core 800mhz dual issue is 1600 x 4 = 6.400 Giga vectors. and a FMAC is 2
already.

each vector is 4 FP32 therefore we are 50 GFLOPs

double the clock rate and 100 GFLOPs is achievable.

however to sustain that we need *SIXTEEN* LDST FunctionUnits.

and each one will have 2x LDST Ports.

32 ports @ around 160 bits wide

that's an internal routing of FIVE THOUSAND wires between the Function
Units and the L0 Cache Buffer.

in addition we will need a minimum of something mad like 8 way striped L1
data caches, and 16x 64 bit Wishbone Buses down into the L2 cache.

it's achievable, it's just going to be a handful.

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Yehowshua]

> that may not be enough to handle framebuffers as well as run an OS.
> framebuffers are scarily demanding of memory bandwidth and they absolutely
> MUST have total absolute top priority.


 The OMI interface has 3200MT/s which is the same spec as DDR4.
**Should be fine depending on line sized which is usually 8 - 64Bytes.

Yehowshua
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Luke Kenneth Casson Leighton]

On Fri, Jun 12, 2020 at 1:38 PM Yehowshua <yimmanuel3@gatech.edu> wrote:
>
>
> > that may not be enough to handle framebuffers as well as run an OS.
> > framebuffers are scarily demanding of memory bandwidth and they absolutely
> > MUST have total absolute top priority.
>
>
>  The OMI interface has 3200MT/s which is the same spec as DDR4.

3200 million transactions/sec, let's assume 8 bytes.  that's 3200
megabytes per second.  25 gigabit / 8 equals 3.125 so that fits with
the math , there.  one 25 gigabit lane provides 3200 million 8-bit
transactions per second (3.125).

4k is 1200 megabytes per second: 37.5% *percent* of the total memory
bandwidth consumed by a 4k 60fps framebuffer!  and if people wanted
120 fps instead, that's now a staggering 75% of a single OMI interface
dedicated to the framebuffer!

bear in mind that that is *just* the framebuffer hardware.  the CPU -
the GPU - is contending for the exact same memory resource.

we cannot possibly expect the GPU to operate effectively at that kind
of resource-starved level.  bear in mind that not only the GPU 3D
Vulkan / bitmap / texturisation data has to go over that same
interface, the Operating System *also* has to swap in and out as well

to get that down to only 5% of total bandwidth, going to 64 bytes per
transaction would, if i am reading this correctly, require *eight* of
those 25 gigabit per second SERDES.

and that alone is going to be mental as far as power consumption is
concerned.  i don't exactly know what the power consumption of a
single lane 25 gigabit SERDES is going to be, i am trying to recall
the conversations last year with Rudi when i visited him: it will
however be extremely important to find out.

i have a *vague* recollection of a number somewhere around 300mW.  if
we assume that to be correct, then those 10x Rx and 14x Tx would be
somewhere around *eight watts* just for the SERDES alone!


so here we have the basis of actionpoints, which i would suggest be
put into the bugtracker (otherwise we'll lose track, which i'm
starting to do, already, thank you gmail)

* to make a decision as to what top level of visual performance is
needed (maximum resolution, maximum framerate)
* calculate the framebuffer data transfer rate for that.
* to decide some top level of GFLOPs and MTriangles/sec (and other
waffly figures for the GPU side)
* estimate the memory bandwidth for the GPU side
* decide if we want to develop a hardware compression algorithm
between memory and framebuffer (an augmentation of Richard Herveille's
RGBTTL framebuffer RTL)
* find out the power consumption in 45nm for these SERDES
* talk to Rudi to see if he can do a SERDES PHY for us
* talk to SymbioticEDA likewise
* talk to Dmitri from LIP6.fr to see what he can do, whether a stable
50 GHz clock is achievable in 45nm (i understand from Dmitri that for
a given target clock - 25 gbit/sec in this case - you need the PLL to
do double the clockrate)

l.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Hendrik Boom]

On Fri, Jun 12, 2020 at 02:30:42PM +0100, Luke Kenneth Casson Leighton wrote:
> On Fri, Jun 12, 2020 at 1:38 PM Yehowshua <yimmanuel3@gatech.edu> wrote:
> >
> >
> > > that may not be enough to handle framebuffers as well as run an OS.
> > > framebuffers are scarily demanding of memory bandwidth and they absolutely
> > > MUST have total absolute top priority.
> >
> >
> >  The OMI interface has 3200MT/s which is the same spec as DDR4.
> 
> 3200 million transactions/sec, let's assume 8 bytes.  that's 3200
> megabytes per second.  25 gigabit / 8 equals 3.125 so that fits with
> the math , there.  one 25 gigabit lane provides 3200 million 8-bit
> transactions per second (3.125).
> 
> 4k is 1200 megabytes per second: 37.5% *percent* of the total memory
> bandwidth consumed by a 4k 60fps framebuffer!  and if people wanted
> 120 fps instead, that's now a staggering 75% of a single OMI interface
> dedicated to the framebuffer!
> 
> bear in mind that that is *just* the framebuffer hardware.  the CPU -
> the GPU - is contending for the exact same memory resource.
> 

It's just possibe that we may not *need* 4k at 120 fps.
Certainly there are many potential applications for our chip that don't 
need that kind of video. 

-- hendrik

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Cole Poirier]

On Jun 12 2020, at 7:06 am, Hendrik Boom <hendrik@topoi.pooq.com> wrote:
> It's just possibe that we may not *need* 4k at 120 fps.
> Certainly there are many potential applications for our chip that
> don't 
> need that kind of video. 

I concur. If we are planning on selling 100 million units, shouldn't we
have several different levels of IO/GPU capablilty, each targeting
different power requirements? For example, a phone or a tablet doesn't
need 4k 120HZ video right?

Cole

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Yehowshua]

> 
> no it really is 25GHz.

Just talked with an engineer who has used the OpenCAPI 
SMC 100 from Microchip. It actually is 25GHz. 25.6 GHz to 
be precise.

Some notes from my conversation with Michael:

>> from the ecp5 note on the serdes
>>   - "3.2 Gbps operation with low 85
>>  mW power per channel”
> 
> Ok
> hmm
> channel i say 1rx or 1tx?
> 
>> 1 tx/rx pair
> 
> 
> so 8 pairs at 3.2Gbps is about a watt
> Can't imagine for 25Gbps
> 
>> Oh, and ecp5 is a 40nm chip
>> so it should be comparable to ours
> 
> 
> that’s what I’m afraid of
> 
> 
>> they might be targeting a different subprocess
>> than us though. I think most fabs offer the 
>> options of like "low power", "high performance”,
>>  or "high density"
> 
>> and I suspect ours will need to be less flexible than the ecp5's so that might save us some power
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Yehowshua]

One of my Create-X mentors pointed me to 
a U.S. organization that can allow us to prototype on 45nm
basically for free.

More on this later when details are finalized.

Yehowshua

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Fri, Jun 12, 2020 at 8:34 PM Yehowshua <yimmanuel3@gatech.edu> wrote:
>
> >
> > no it really is 25GHz.
>
> Just talked with an engineer who has used the OpenCAPI
> SMC 100 from Microchip. It actually is 25GHz. 25.6 GHz to
> be precise.

makes sense, maps to 3200 gbytes/sec, which is the speeds you see for
things like the latest NVMe

l.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Jacob Lifshay]

On Fri, Jun 12, 2020 at 2:43 PM Luke Kenneth Casson Leighton
<lkcl@lkcl.net> wrote:
>
> ---
> crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
>
> On Fri, Jun 12, 2020 at 8:34 PM Yehowshua <yimmanuel3@gatech.edu> wrote:
> >
> > >
> > > no it really is 25GHz.
> >
> > Just talked with an engineer who has used the OpenCAPI
> > SMC 100 from Microchip. It actually is 25GHz. 25.6 GHz to
> > be precise.
>
> makes sense, maps to 3200 gbytes/sec, which is the speeds you see for
> things like the latest NVMe

Wow! 3.2 hard drives per second! :P

(I'm assuming you meant 3.2GB/s not 3.2TB/s)
>
> l.
>
> _______________________________________________
> libre-riscv-dev mailing list
> libre-riscv-dev@lists.libre-riscv.org
> http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] Fall 2022 Interfaces

[Luke Kenneth Casson Leighton]

On Saturday, June 13, 2020, Jacob Lifshay <programmerjake@gmail.com> wrote:

>
> > makes sense, maps to 3200 gbytes/sec, which is the speeds you see for
> > things like the latest NVMe
>
> Wow! 3.2 hard drives per second! :P
>
> (I'm assuming you meant 3.2GB/s not 3.2TB/s)


haha, yes :)



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev