[libre-riscv-dev] system call (sc) LEV "reserved field"

[libre-riscv-dev] system call (sc) LEV "reserved field"

[Luke Kenneth Casson Leighton]

hi, we're just reviewing the behaviour needed when LEV != 0, and are
following what microwatt does (which does not have hypervisor support
yet)

https://bugs.libre-soc.org/show_bug.cgi?id=325#c106

so the trail - i am so glad that the PDF has cross-reference linking -
jumps from one section to another and after jumping 5 times we
eventually ascertain the hypothesis that reserved fields, if set,
should raise an "illegal instruction".

however this is so unclear (because of the referral from one section
to another) that i am seeking confirmation.  should we raise an
"illegal instruction" when "LEV > 1" on sc?

secondly, we note that "LEV=1" is for invocation of the hypervisor.
what's not clear to us is - given that we are not implementing
hypervisor - should this be *also* treated as an illegal instruction?
or, should we just leave it to fall through to trap @ addr 0x0c00, and
expect the trap *there* to notice and deal with the situation?

also: if we set the HV bit in MSR (when LEV=1) section 6.5.14 p1077
which refers us back to figure 65 on p1064, will this "break" things?

also: in microwatt, i'm not seeing the remaining bits which appear [to
need to] be set.

https://github.com/antonblanchard/microwatt/blob/master/execute1.vhdl#L479
            ctrl_tmp.msr(MSR_SF) <= '1';
            ctrl_tmp.msr(MSR_EE) <= '0';
            ctrl_tmp.msr(MSR_PR) <= '0';
            ctrl_tmp.msr(MSR_IR) <= '0';
            ctrl_tmp.msr(MSR_DR) <= '0';
            ctrl_tmp.msr(MSR_RI) <= '0';
            ctrl_tmp.msr(MSR_LE) <= '1';

these appear to be correct as defined according to figure 65 (p1063)

however the remaining actions do not seem to be implemented (p1064):

     Bits bit 5, TM, VEC, VSX, PR, FP, and PMM are set to 0.
     The TE field is set to 0b00.
     TM, FP, VEC, VSX, and bit 5 are set to 0.

question: what effect would it have - bear in mind that we are
following microwatt - if we implemented these changes to MSR?  bear in
mind that we ignore most of them at the moment (MSR.LE being one
notable exception), so the question is, in effect: does the Linux
kernel *also* ignore them?

tia,

l.

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] [OpenPOWER-HDL-Cores] system call (sc) LEV "reserved field"

[Paul Mackerras]

On Wed, Jul 22, 2020 at 03:22:43PM +0100, Luke Kenneth Casson Leighton wrote:
> hi, we're just reviewing the behaviour needed when LEV != 0, and are
> following what microwatt does (which does not have hypervisor support
> yet)
> 
> https://bugs.libre-soc.org/show_bug.cgi?id=325#c106
> 
> so the trail - i am so glad that the PDF has cross-reference linking -
> jumps from one section to another and after jumping 5 times we
> eventually ascertain the hypothesis that reserved fields, if set,
> should raise an "illegal instruction".

The first sentence of Book I section 1.3.3 says "Reserved fields in
instructions are ignored by the processor."  What led you to confirm
your hypothesis that reserved bits being set should cause an illegal
instruction interrupt?

> however this is so unclear (because of the referral from one section
> to another) that i am seeking confirmation.  should we raise an
> "illegal instruction" when "LEV > 1" on sc?

Section 1.8.2 (Book I) says "any attempt to execute an invalid form of
an instruction will either cause the system illegal instruction
handler to be invoked or yield boundedly undefined results".  Putting
LEV=1 in sc would be an example of an invalid form (on an
implementation without hypervisor mode).  A boundedly undefined result
is one which could be obtained by a sequence of valid instructions,
so in the case of sc 1, making it do what sc 0 does meets the
boundedly undefined results requirement.

> secondly, we note that "LEV=1" is for invocation of the hypervisor.
> what's not clear to us is - given that we are not implementing
> hypervisor - should this be *also* treated as an illegal instruction?
> or, should we just leave it to fall through to trap @ addr 0x0c00, and
> expect the trap *there* to notice and deal with the situation?

That is what I would do.

There is one of the variants of KVM on PPC, called KVM-PR, which runs
the guest entirely in user mode and traps and emulates all privileged
instructions (thus it doesn't need hypervisor mode and can run inside
a guest of another hypervisor).  If you are running a KVM guest inside
that environment and the guest does sc 1, KVM-PR expects that to end
up at the kernel's 0xc00 handler.  So that is one reason to treat sc 1
as sc 0.

> also: if we set the HV bit in MSR (when LEV=1) section 6.5.14 p1077
> which refers us back to figure 65 on p1064, will this "break" things?

Probably not.  Linux does check whether HV=1 at boot time, but I'm
pretty sure that's only on certain processors which it knows to be
HV-capable (either by looking at PVR or the device tree).

> also: in microwatt, i'm not seeing the remaining bits which appear [to
> need to] be set.
> 
> https://github.com/antonblanchard/microwatt/blob/master/execute1.vhdl#L479
>             ctrl_tmp.msr(MSR_SF) <= '1';
>             ctrl_tmp.msr(MSR_EE) <= '0';
>             ctrl_tmp.msr(MSR_PR) <= '0';
>             ctrl_tmp.msr(MSR_IR) <= '0';
>             ctrl_tmp.msr(MSR_DR) <= '0';
>             ctrl_tmp.msr(MSR_RI) <= '0';
>             ctrl_tmp.msr(MSR_LE) <= '1';
> 
> these appear to be correct as defined according to figure 65 (p1063)
> 
> however the remaining actions do not seem to be implemented (p1064):
> 
>      Bits bit 5, TM, VEC, VSX, PR, FP, and PMM are set to 0.
>      The TE field is set to 0b00.
>      TM, FP, VEC, VSX, and bit 5 are set to 0.

Right.  We have a to-do list for architecture compliance.  (We haven't
implemented 32-bit mode or BE mode, for instance.)

> question: what effect would it have - bear in mind that we are
> following microwatt - if we implemented these changes to MSR?  bear in
> mind that we ignore most of them at the moment (MSR.LE being one
> notable exception), so the question is, in effect: does the Linux
> kernel *also* ignore them?

The Linux kernel clearly needs PR to be set to zero and it also
expects FP, VEC, VSX, TM to be cleared.  Setting TE to 0 is necessary
once you implement the trace interrupt, otherwise you could get a
trace interrupt inside your first-level interrupt handlers, which
would be bad.  Similarly if you have floating-point and you don't set
FE0 and FE1 to 0 on an interrupt, there is the chance of taking a
floating-point program interrupt inside a first-level handler.

I'm not sure that all this counts as the Linux kernel "ignoring" the
bits, but in general if you do what the architecture says, the kernel
will be happier than if you don't.

Regards,
Paul.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] system call (sc) LEV "reserved field"

[Luke Kenneth Casson Leighton]

On Wednesday, July 22, 2020, Paul Mackerras <paulus@ozlabs.org> wrote:

> On Wed, Jul 22, 2020 at 03:22:43PM +0100, Luke Kenneth Casson Leighton
> wrote:
> > hi, we're just reviewing the behaviour needed when LEV != 0, and are
> > following what microwatt does (which does not have hypervisor support
> > yet)
> >
> > https://bugs.libre-soc.org/show_bug.cgi?id=325#c106
> >
> > so the trail - i am so glad that the PDF has cross-reference linking -
> > jumps from one section to another and after jumping 5 times we
> > eventually ascertain the hypothesis that reserved fields, if set,
> > should raise an "illegal instruction".
>
> The first sentence of Book I section 1.3.3 says "Reserved fields in
> instructions are ignored by the processor."  What led you to confirm
> your hypothesis that reserved bits being set should cause an illegal
> instruction interrupt?


because i did not expect that behaviour, because doing so (ignoring them)
makes it impossible to trap and emulate.  (it becomes necessary to use JIT
analysis)

so, when some bit is added in the future, an older processor (and the
device it is in) basically has to be thrown into landfill.

if however reserved bits being set cause an exception, the "old" processor
stands a chance of emulating the new behaviour (in software, even if that's
slow), giving it a chance of keeping out of landfill for slightly longer.

however it is not appropriate for all systems to raise exceptions on
reserved bits: the cost of having the detection hardware (a full POWER9
decoder and also illegal/unsupported/reserved SPR detection) can be very
high especially for resource and power constrained silicon or FPGAs.

(example: i know someone - yea, you Sam - who implemented RV64 to comply
with the UNIX RISCV spec rather than the Embedded RISCV spec: the "CSR
detection" just to support all the zeros and illegal CSRs took a whopping
15% of an ICE40 FPGA!)

in RISC-V they get this right, by having two separate Platforms:

* Embedded which is permitted to ignore reserved bits entirely

* UNIX, which definitely is not.

for Embedded, the vendor customises the firmware entirely, and binary
interoperability as well as legacy software support is completely
unimportant.

for UNIXen, interoperability and longterm stability we know very well is
critical.

bottom line if it is correct that on the PowerISA UNIX Platform reserved
bits can be ignored that is cause for some concern, where for Embedded it
would be the other way round: cause for concern if the reserved bits could
*not* be ignored.



> > however this is so unclear (because of the referral from one section
> > to another) that i am seeking confirmation.  should we raise an
> > "illegal instruction" when "LEV > 1" on sc?
>
> Section 1.8.2 (Book I) says "any attempt to execute an invalid form of
> an instruction will either cause the system illegal instruction
> handler to be invoked or yield boundedly undefined results".  Putting
> LEV=1 in sc would be an example of an invalid form (on an
> implementation without hypervisor mode).


ok that helps clarify what that means, thank you.


>   A boundedly undefined result
> is one which could be obtained by a sequence of valid instructions,
> so in the case of sc 1, making it do what sc 0 does meets the
> boundedly undefined results requirement.


ok so that... if i am understanding correctly, means, "you can in fact do
something different and OS software has to detect it and sort it out to
yield expected behaviour"

which, if i am being honest, makes me nervous :)


> > secondly, we note that "LEV=1" is for invocation of the hypervisor.
> > what's not clear to us is - given that we are not implementing
> > hypervisor - should this be *also* treated as an illegal instruction?
> > or, should we just leave it to fall through to trap @ addr 0x0c00, and
> > expect the trap *there* to notice and deal with the situation?
>
> That is what I would do.


ok.  we can do that.


> There is one of the variants of KVM on PPC, called KVM-PR, which runs
> the guest entirely in user mode and traps and emulates all privileged
> instructions (thus it doesn't need hypervisor mode and can run inside
> a guest of another hypervisor).  If you are running a KVM guest inside
> that environment and the guest does sc 1, KVM-PR expects that to end
> up at the kernel's 0xc00 handler.  So that is one reason to treat sc 1
> as sc 0.


ahh.  i did wonder :)


>
> > also: if we set the HV bit in MSR (when LEV=1) section 6.5.14 p1077
> > which refers us back to figure 65 on p1064, will this "break" things?
>
> Probably not.  Linux does check whether HV=1 at boot time, but I'm
> pretty sure that's only on certain processors which it knows to be
> HV-capable (either by looking at PVR or the device tree).


ok.  thank you.


>
> > also: in microwatt, i'm not seeing the remaining bits which appear [to
> > need to] be set.
> >
> > https://github.com/antonblanchard/microwatt/blob/
> master/execute1.vhdl#L479
> >             ctrl_tmp.msr(MSR_SF) <= '1';
> >             ctrl_tmp.msr(MSR_EE) <= '0';
> >             ctrl_tmp.msr(MSR_PR) <= '0';
> >             ctrl_tmp.msr(MSR_IR) <= '0';
> >             ctrl_tmp.msr(MSR_DR) <= '0';
> >             ctrl_tmp.msr(MSR_RI) <= '0';
> >             ctrl_tmp.msr(MSR_LE) <= '1';
> >
> > these appear to be correct as defined according to figure 65 (p1063)
> >
> > however the remaining actions do not seem to be implemented (p1064):
> >
> >      Bits bit 5, TM, VEC, VSX, PR, FP, and PMM are set to 0.
> >      The TE field is set to 0b00.
> >      TM, FP, VEC, VSX, and bit 5 are set to 0.
>
> Right.  We have a to-do list for architecture compliance.  (We haven't
> implemented 32-bit mode or BE mode, for instance.)


yeahh although we have 32 bit op modes (using microwatt  decode1.vhdl,
turned into CSV) we have yet to support the MSR 32bit global mode.

LE/BE amazingly seems to work on LibreSOC, it was quite funny having the
trap jump into 0x700 when testing against qemu (running singlestep under
gdb), only to find that qemu traps change the LE bit and of course in qemu
once that's changed gdb can't read registers correctly. sigh.


> > question: what effect would it have - bear in mind that we are
> > following microwatt - if we implemented these changes to MSR?  bear in
> > mind that we ignore most of them at the moment (MSR.LE being one
> > notable exception), so the question is, in effect: does the Linux
> > kernel *also* ignore them?
>
> The Linux kernel clearly needs PR to be set to zero and it also
> expects FP, VEC, VSX, TM to be cleared.  Setting TE to 0 is necessary
> once you implement the trace interrupt, otherwise you could get a
> trace interrupt inside your first-level interrupt handlers, which
> would be bad.


ah :)


>  Similarly if you have floating-point and you don't set
> FE0 and FE1 to 0 on an interrupt, there is the chance of taking a
> floating-point program interrupt inside a first-level handler.


whoops.  ok appreciate the warning.


>
> I'm not sure that all this counts as the Linux kernel "ignoring" the
> bits, but in general if you do what the architecture says, the kernel
> will be happier than if you don't.


ha, that makes sense.

i generally found this out when network reverse-engineering, despite not
understanding at all what i was sending to the client or server :)

thank you Paul

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] [OpenPOWER-HDL-Cores] system call (sc) LEV "reserved field"

[Benjamin Herrenschmidt]

On Thu, 2020-07-23 at 00:41 +0100, Luke Kenneth Casson Leighton wrote:
> 
> because i did not expect that behaviour, because doing so (ignoring
> them) makes it impossible to trap and emulate.  (it becomes necessary
> to use JIT analysis)
> 
> so, when some bit is added in the future, an older processor (and the
> device it is in) basically has to be thrown into landfill.

Depends, it goes both ways. For example, this is what allowed the
addition of lwsync without throwing older processors into the landfill
because it would automatically "escalate" to a full sync on processors
that didn't implement it (or should have... FSL did screw that up on
some cores).
 
> if however reserved bits being set cause an exception, the "old"
> processor stands a chance of emulating the new behaviour (in
> software, even if that's slow), giving it a chance of keeping out of
> landfill for slightly longer.

Which is why powerpc tends not to "add bits" to instructions unless
ignoring them is a safe fallback.

> however it is not appropriate for all systems to raise exceptions on
> reserved bits: the cost of having the detection hardware (a full
> POWER9 decoder and also illegal/unsupported/reserved SPR detection)
> can be very high especially for resource and power constrained
> silicon or FPGAs.
> 
> (example: i know someone - yea, you Sam - who implemented RV64 to
> comply with the UNIX RISCV spec rather than the Embedded RISCV spec:
> the "CSR detection" just to support all the zeros and illegal CSRs
> took a whopping 15% of an ICE40 FPGA!)
> 
> in RISC-V they get this right, by having two separate Platforms:
> 
> * Embedded which is permitted to ignore reserved bits entirely
> 
> * UNIX, which definitely is not.
> 
> for Embedded, the vendor customises the firmware entirely, and binary
> interoperability as well as legacy software support is completely
> unimportant.
> 
> for UNIXen, interoperability and longterm stability we know very well
> is critical.
> 
> bottom line if it is correct that on the PowerISA UNIX Platform
> reserved bits can be ignored that is cause for some concern, where
> for Embedded it would be the other way round: cause for concern if
> the reserved bits could *not* be ignored.

Do you have more specific concerns here ? IE. Actual examples where
this has been cause of breakage in the past ?

> > > however this is so unclear (because of the referral from one
> > section
> > > to another) that i am seeking confirmation.  should we raise an
> > > "illegal instruction" when "LEV > 1" on sc?
> > 
> > Section 1.8.2 (Book I) says "any attempt to execute an invalid form
> > of
> > an instruction will either cause the system illegal instruction
> > handler to be invoked or yield boundedly undefined results". 
> > Putting
> > LEV=1 in sc would be an example of an invalid form (on an
> > implementation without hypervisor mode).
> 
> ok that helps clarify what that means, thank you.
>  
> >   A boundedly undefined result
> > is one which could be obtained by a sequence of valid instructions,
> > so in the case of sc 1, making it do what sc 0 does meets the
> > boundedly undefined results requirement.
> 
> ok so that... if i am understanding correctly, means, "you can in
> fact do something different and OS software has to detect it and sort
> it out to yield expected behaviour"
>  
> which, if i am being honest, makes me nervous :)
> 
> > > secondly, we note that "LEV=1" is for invocation of the
> > hypervisor.
> > > what's not clear to us is - given that we are not implementing
> > > hypervisor - should this be *also* treated as an illegal
> > instruction?
> > > or, should we just leave it to fall through to trap @ addr
> > 0x0c00, and
> > > expect the trap *there* to notice and deal with the situation?
> > 
> > That is what I would do.
> 
> ok.  we can do that.
>  
> > There is one of the variants of KVM on PPC, called KVM-PR, which
> > runs
> > the guest entirely in user mode and traps and emulates all
> > privileged
> > instructions (thus it doesn't need hypervisor mode and can run
> > inside
> > a guest of another hypervisor).  If you are running a KVM guest
> > inside
> > that environment and the guest does sc 1, KVM-PR expects that to
> > end
> > up at the kernel's 0xc00 handler.  So that is one reason to treat
> > sc 1
> > as sc 0.
> 
> ahh.  i did wonder :)
>  
> > > also: if we set the HV bit in MSR (when LEV=1) section 6.5.14
> > p1077
> > > which refers us back to figure 65 on p1064, will this "break"
> > things?
> > 
> > Probably not.  Linux does check whether HV=1 at boot time, but I'm
> > pretty sure that's only on certain processors which it knows to be
> > HV-capable (either by looking at PVR or the device tree).
> 
> ok.  thank you.
>  
> > > also: in microwatt, i'm not seeing the remaining bits which
> > appear [to
> > > need to] be set.
> > > 
> > > 
> > https://github.com/antonblanchard/microwatt/blob/master/execute1.vhdl#L479
> > >             ctrl_tmp.msr(MSR_SF) <= '1';
> > >             ctrl_tmp.msr(MSR_EE) <= '0';
> > >             ctrl_tmp.msr(MSR_PR) <= '0';
> > >             ctrl_tmp.msr(MSR_IR) <= '0';
> > >             ctrl_tmp.msr(MSR_DR) <= '0';
> > >             ctrl_tmp.msr(MSR_RI) <= '0';
> > >             ctrl_tmp.msr(MSR_LE) <= '1';
> > > 
> > > these appear to be correct as defined according to figure 65
> > (p1063)
> > > 
> > > however the remaining actions do not seem to be implemented
> > (p1064):
> > > 
> > >      Bits bit 5, TM, VEC, VSX, PR, FP, and PMM are set to 0.
> > >      The TE field is set to 0b00.
> > >      TM, FP, VEC, VSX, and bit 5 are set to 0.
> > 
> > Right.  We have a to-do list for architecture compliance.  (We
> > haven't
> > implemented 32-bit mode or BE mode, for instance.)
> 
> yeahh although we have 32 bit op modes (using microwatt 
> decode1.vhdl, turned into CSV) we have yet to support the MSR 32bit
> global mode.
> 
> LE/BE amazingly seems to work on LibreSOC, it was quite funny having
> the trap jump into 0x700 when testing against qemu (running
> singlestep under gdb), only to find that qemu traps change the LE bit
> and of course in qemu once that's changed gdb can't read registers
> correctly. sigh.

It can if you manually change endian in gdb no ?

> > > question: what effect would it have - bear in mind that we are
> > > following microwatt - if we implemented these changes to MSR? 
> > bear in
> > > mind that we ignore most of them at the moment (MSR.LE being one
> > > notable exception), so the question is, in effect: does the Linux
> > > kernel *also* ignore them?
> > 
> > The Linux kernel clearly needs PR to be set to zero and it also
> > expects FP, VEC, VSX, TM to be cleared.  Setting TE to 0 is
> > necessary
> > once you implement the trace interrupt, otherwise you could get a
> > trace interrupt inside your first-level interrupt handlers, which
> > would be bad.
> 
> ah :)
>  
> >  Similarly if you have floating-point and you don't set
> > FE0 and FE1 to 0 on an interrupt, there is the chance of taking a
> > floating-point program interrupt inside a first-level handler.
> 
> whoops.  ok appreciate the warning.
>  
> > I'm not sure that all this counts as the Linux kernel "ignoring"
> > the
> > bits, but in general if you do what the architecture says, the
> > kernel
> > will be happier than if you don't.
> 
> ha, that makes sense.
> 
> i generally found this out when network reverse-engineering, despite
> not understanding at all what i was sending to the client or server
> :)
> 
> thank you Paul
> 
> l.
> 
> 
> 
> _______________________________________________
> OpenPOWER-HDL-Cores mailing list
> OpenPOWER-HDL-Cores@mailinglist.openpowerfoundation.org
> 
http://lists.mailinglist.openpowerfoundation.org/mailman/listinfo/openpower-hdl-cores


_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] [OpenPOWER-HDL-Cores] system call (sc) LEV "reserved field"

[Paul Mackerras]

On Thu, Jul 23, 2020 at 12:41:15AM +0100, Luke Kenneth Casson Leighton wrote:
> On Wednesday, July 22, 2020, Paul Mackerras <paulus@ozlabs.org> wrote:
> 
> > On Wed, Jul 22, 2020 at 03:22:43PM +0100, Luke Kenneth Casson Leighton
> > wrote:
> > > hi, we're just reviewing the behaviour needed when LEV != 0, and are
> > > following what microwatt does (which does not have hypervisor support
> > > yet)
> > >
> > > https://bugs.libre-soc.org/show_bug.cgi?id=325#c106
> > >
> > > so the trail - i am so glad that the PDF has cross-reference linking -
> > > jumps from one section to another and after jumping 5 times we
> > > eventually ascertain the hypothesis that reserved fields, if set,
> > > should raise an "illegal instruction".
> >
> > The first sentence of Book I section 1.3.3 says "Reserved fields in
> > instructions are ignored by the processor."  What led you to confirm
> > your hypothesis that reserved bits being set should cause an illegal
> > instruction interrupt?
> 
> 
> because i did not expect that behaviour, because doing so (ignoring them)
> makes it impossible to trap and emulate.  (it becomes necessary to use JIT
> analysis)
> 
> so, when some bit is added in the future, an older processor (and the
> device it is in) basically has to be thrown into landfill.

No, it depends on what the new bits do.  Generally when a previously
reserved field in an existing instruction is defined to have a
meaning, it is done in such a way that the behaviour of old processors
is an acceptable implementation of the instruction for any value of
the new field (though perhaps not the most performant).  Two examples:

1. The sync instruction had a reserved field which later got defined
as the "L" field for the instruction, with L=1 defined as light-weight
sync.  Light-weight sync does less than the normal sync and runs
faster.  Old processors that ignore the L field do the full normal
sync, which is acceptable if you are asking for the light-weight sync.

2. When the mfocrf (move from one CR field) instruction was created,
the architects deliberately used the same major/minor opcode as mfcr
(move from CR) and defined mfocrf in such a way that the mfcr
behaviour would be an acceptable implementation of mfocrf.
Unfortunately it turned out that the hardware designers of the
previous generation processor (which was POWER3) had put in a check
that reserved instruction fields had to be zero, which then meant that
software couldn't just use mfocrf everywhere, but had to be careful
to use mfcr rather than mfocrf on POWER3.

I agree that if a previously reserved field was given a meaning that
completely changed an instruction's behaviour, that would cause real
problems, but the Power ISA architects have not done that.  In cases
where significantly different or new behaviour is needed, a new major
or minor opcode would be assigned.  Undefined opcodes definitely do
cause an illegal instruction interrupt.

> if however reserved bits being set cause an exception, the "old" processor
> stands a chance of emulating the new behaviour (in software, even if that's
> slow), giving it a chance of keeping out of landfill for slightly longer.

The cost here is that you have broken compatibility for older system
software versions that don't understand the reason for the illegal
instruction interrupt.

> however it is not appropriate for all systems to raise exceptions on
> reserved bits: the cost of having the detection hardware (a full POWER9
> decoder and also illegal/unsupported/reserved SPR detection) can be very
> high especially for resource and power constrained silicon or FPGAs.
> 
> (example: i know someone - yea, you Sam - who implemented RV64 to comply
> with the UNIX RISCV spec rather than the Embedded RISCV spec: the "CSR
> detection" just to support all the zeros and illegal CSRs took a whopping
> 15% of an ICE40 FPGA!)
> 
> in RISC-V they get this right, by having two separate Platforms:
> 
> * Embedded which is permitted to ignore reserved bits entirely
> 
> * UNIX, which definitely is not.
> 
> for Embedded, the vendor customises the firmware entirely, and binary
> interoperability as well as legacy software support is completely
> unimportant.
> 
> for UNIXen, interoperability and longterm stability we know very well is
> critical.

Which has been a critical consideration for Power server processors
over the last 25 years, and has been achieved by discipline on how new
things get architected rather than by emulation.

> bottom line if it is correct that on the PowerISA UNIX Platform reserved
> bits can be ignored that is cause for some concern, where for Embedded it
> would be the other way round: cause for concern if the reserved bits could
> *not* be ignored.
> 
> 
> 
> > > however this is so unclear (because of the referral from one section
> > > to another) that i am seeking confirmation.  should we raise an
> > > "illegal instruction" when "LEV > 1" on sc?
> >
> > Section 1.8.2 (Book I) says "any attempt to execute an invalid form of
> > an instruction will either cause the system illegal instruction
> > handler to be invoked or yield boundedly undefined results".  Putting
> > LEV=1 in sc would be an example of an invalid form (on an
> > implementation without hypervisor mode).
> 
> 
> ok that helps clarify what that means, thank you.
> 
> 
> >   A boundedly undefined result
> > is one which could be obtained by a sequence of valid instructions,
> > so in the case of sc 1, making it do what sc 0 does meets the
> > boundedly undefined results requirement.
> 
> 
> ok so that... if i am understanding correctly, means, "you can in fact do
> something different and OS software has to detect it and sort it out to
> yield expected behaviour"
> 
> which, if i am being honest, makes me nervous :)

There is of course a quality of implementation question here - this
isn't really a licence for hardware designers to do any crazy thing
they want.  I think it would be impossibly unwieldy for the
architecture to try to define in black-letter law what constitutes
"reasonable" vs. "crazy" behaviour for the undefined cases.  By saying
"boundedly" undefined they are at least saying that there must be no
way to escape privilege boundaries by exploiting undefined behaviour.
Beyond that it's up to the hardware designers to do something
sensible.

Regards,
Paul.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] system call (sc) LEV "reserved field"

[Luke Kenneth Casson Leighton]

hi benjamin, welcome, and thank you for your insights.

On Thursday, July 23, 2020, Benjamin Herrenschmidt <benh@kernel.crashing.org>
wrote:

> On Thu, 2020-07-23 at 00:41 +0100, Luke Kenneth Casson Leighton wrote:
>
> > so, when some bit is added in the future, an older processor (and the
> > device it is in) basically has to be thrown into landfill.
>
> Depends, it goes both ways. For example, this is what allowed the
> addition of lwsync without throwing older processors into the landfill
> because it would automatically "escalate" to a full sync on processors
> that didn't implement it


ah.   on first glance this seems ok.


> (or should have... FSL did screw that up on
> some cores).


oh whoops :)

what i have observed (again this is from RISCV), for instructions that can
be "safely ignored" are of a "hint" type.

the RISCV Fence instructions also fell into this category.  being
optimisations, less complex implementations could completely ignore them.

however in the example that you give - lwsync - i take it that this is a
"less costly" cache/sync flush type instruction than the one that it was
added to?

i.e. old sync is a "superset" of lwsync?

this is very interesting to me, because i have made a longterm study of
"how to develop stable specifications", and it is the first time in 25
years that i have encountered a counterexample to the practice of
disallowing "reserved behaviour".

the "normal" way in "reserved is illegal" specifications, would be that
lwsync would raise an illegal instruction on a processor that did not
support it, the author of the trap would know that the extra bit did not
matter, the trap would call the alternative sync instruction, and return
from the trap.

i do not know the full story about what FSL is: without knowing the details
i suspect that a trap *could* have sorted things out, there, *if* the older
processor was required to raise "illegal instruction"

but because the bit is ignored, there is no way that can be done.

so whilst on the face of it, lwsync *sounds* like a counterexample, instead
(caveat, i do not know what FSL is), it seems more to support the case
*for* use of reserved bits raising illegal instructions [on UNIX platforms]

where of course this would become problematic is if lwsync (or other future
added instruction) was part of a tight, performance-critical loop.  a trap
and emulate in the middle of that would severely degrade older processor
performance.  there is nothing that can be done about that.


> if however reserved bits being set cause an exception, the "old"
> > processor stands a chance of emulating the new behaviour (in
> > software, even if that's slow), giving it a chance of keeping out of
> > landfill for slightly longer.
>
> Which is why powerpc tends not to "add bits" to instructions unless
> ignoring them is a safe fallback.


as we can see above, the example that was given  shows that this is a
problematic approach.

it *sounds* safe to "downgrade" an instruction that does more work (more
syncing) to one that does less, however this is a pretty unique case.

i would say that PowerISA "should have had" space for such downgrading
already planned in advance, except, ruefully and regretfully, such is not
possible or practical (changing the past) and pragmatically we have to work
with how things are, now.

use of "hints" would not help here either unless they were a macro-op fused
type of hint.

wait... that would work.

a sequence as follows:

* hint instruction "next instruction is to be lwsync"
* old sync instruction

older processors would:

* ignore the hint
* execute the old sync instruction as-is, safely doing more work than
necessary.

newer processors would macro-op fuse the two into an lwsync.

of course now with prefixes in POWER10 / v3.1, prefixes could do the same
job without running out of hint space very rapidly.

and Opcode 1 is of course "illegal instruction", and could be used
extensively for "instruction upgrading"

this would work really well, leaving hint macro-op fusion for those very
special cases similar to lwsync where "behaviour downgrading" is needed.

hint macro-op fusion would also work really well in performance critical
loops, too.


> > bottom line if it is correct that on the PowerISA UNIX Platform
> > reserved bits can be ignored that is cause for some concern, where
> > for Embedded it would be the other way round: cause for concern if
> > the reserved bits could *not* be ignored.
>
> Do you have more specific concerns here ?


sorry, Benjamin, i apologise for saying this: the question appears to be
implying, very subtly, that you've not taken on board what i've said. if
you had summarised what i wrote, in your own words, i would have a clear
indicator that you'd heard me. if that is a misinterpretation i apologise.

the specific concern *is* that reserved bits do not cause illegal
instructions to be raised on the UNIX Platform, because without that in
place, UNIX platform longterm seamless upgradeability and interoperability
between vendors across all their product ranges over time is simply not
possible.

i have not spelled it out explicitly, seeking instead to use logical
reasoning rather than make categorical statements, but this is actually a
really serious problem.


IE. Actual examples where
> this has been cause of breakage in the past ?


with PowerISA having only a few implementors, even though i am new to
PowerISA i do not expect there to be many such examples, simply because as
i said, for he "Embedded" class processors (Motorola's 32 bit range, and
even the Quorl 64 bit range can be said to be targetted at the "Embedded"
market of Networking and Routing), these will not encounter this as a
problem (because of the customised firmware that they run).

it is only the *UNIX* Platform where this matters.

that leaves, realistically, only the IBM POWER range.  and when there is
only one vendor, interoperability with other vendors is not important when
there aren't any other vendors!

however if PowerISA expands (as we know it would like to), and this is not
changed, i *guarantee* that when more implementors are involved there will
be problems, here.

in other words the approach of ignoring reserved bits has only worked (for
the past 20 years?) because IBM, realistically, is the only (non-Embedded)
implementor of UNIX Platform processors.


> LE/BE amazingly seems to work on LibreSOC, it was quite funny having
> > the trap jump into 0x700 when testing against qemu (running
> > singlestep under gdb), only to find that qemu traps change the LE bit
> > and of course in qemu once that's changed gdb can't read registers
> > correctly. sigh.
>
> It can if you manually change endian in gdb no ?


this (for a geek) is why it is amusing, because we did exactly that.

during setup, i call the machine-interface "set endian" command depending
on whether the program being executed is LE or BE.

breakpoints are set on 0x700 and end of program.

all good so far

then the trap occurs, and the MSR changes at the trap breakpoint.

when we read the PC it is 0x0070000000000000 not 0x700

when we read the MSR it is *also* byte reversed.

so there is no way to read the MSR and determine from the bits returned if
the bits need to be reversed because the bits that tell you you should
reverse the bits are already reversed.

doh :)

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] system call (sc) LEV "reserved field"

[Luke Kenneth Casson Leighton]

On Thursday, July 23, 2020, Paul Mackerras <paulus@ozlabs.org> wrote:

>
>
> No, it depends on what the new bits do.


hi paul, bit of crossover (replying to ben), v. late here (early) will
catch up tomorrow.

best,

l.




-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] system call (sc) LEV "reserved field"

[Benjamin Herrenschmidt]

On Thu, 2020-07-23 at 05:16 +0100, Luke Kenneth Casson Leighton wrote:
> 
> what i have observed (again this is from RISCV), for instructions
> that can be "safely ignored" are of a "hint" type.

See what Paul said... the idea is that any change to an existing
instruction that makes use of formerly reserved bits needs to be done
in such a way that the ignoring the bits is a correct implementation.

In practice that often means the new bits are indeed hints, but not
always, see Paul example about mfocrf.

> the RISCV Fence instructions also fell into this category.  being
> optimisations, less complex implementations could completely ignore
> them.
> 
> however in the example that you give - lwsync - i take it that this
> is a "less costly" cache/sync flush type instruction than the one
> that it was added to?
> 
> i.e. old sync is a "superset" of lwsync?

sync is a memory barrier. lwsync is a lighter weight version of it, so
yes, sync is a superset of lwsync.

> this is very interesting to me, because i have made a longterm study
> of "how to develop stable specifications", and it is the first time
> in 25 years that i have encountered a counterexample to the practice
> of disallowing "reserved behaviour".
> 
> the "normal" way in "reserved is illegal" specifications, would be
> that lwsync would raise an illegal instruction on a processor that
> did not support it, the author of the trap would know that the extra
> bit did not matter, the trap would call the alternative sync
> instruction, and return from the trap.

The performance impact of this would make it entirely unusable however.

> i do not know the full story about what FSL is: without knowing the
> details i suspect that a trap *could* have sorted things out, there,
> *if* the older processor was required to raise "illegal instruction"

FSL is Freescale. Some variants of their embedded cores did trap, which
forced the kernel to do some nasty dynamic patching at runtime to get
rid of lwsync's on these. The cost of a trap is way too prohibitive
(and in some cases those could be use in code path that arent safe to
take traps in).

> but because the bit is ignored, there is no way that can be done.

But that's fine because "sync" is a strict superset so ignoring the bit
works just fine.

> so whilst on the face of it, lwsync *sounds* like a counterexample,
> instead (caveat, i do not know what FSL is), it seems more to support
> the case *for* use of reserved bits raising illegal instructions [on
> UNIX platforms]

I fail to follow your logic. The trap case is a complete non-starter
for an instruction meant to be a lightweight memory barrier. The
transparent "fallback" to a full memory barrier on the contrary works
fine. There is never any case where "trapping an emulating" is the
desired behaviour.

> where of course this would become problematic is if lwsync (or other
> future added instruction) was part of a tight, performance-critical
> loop.  a trap and emulate in the middle of that would severely
> degrade older processor performance.  there is nothing that can be
> done about that.

Precisely.
> 
> > > if however reserved bits being set cause an exception, the "old"
> > > processor stands a chance of emulating the new behaviour (in
> > > software, even if that's slow), giving it a chance of keeping out
> > of
> > > landfill for slightly longer.
> > 
> > Which is why powerpc tends not to "add bits" to instructions unless
> > ignoring them is a safe fallback.
> 
> as we can see above, the example that was given  shows that this is a
> problematic approach.

I fail to understand the "problem" in your example, maybe you can
rephrase it ?

 .../...

> sorry, Benjamin, i apologise for saying this: the question appears to
> be implying, very subtly, that you've not taken on board what i've
> said. if you had summarised what i wrote, in your own words, i would
> have a clear indicator that you'd heard me. if that is a
> misinterpretation i apologise.
> 
> the specific concern *is* that reserved bits do not cause illegal
> instructions to be raised on the UNIX Platform, because without that
> in place, UNIX platform longterm seamless upgradeability and
> interoperability between vendors across all their product ranges over
> time is simply not possible.

Experience with decades of POWER architecture however seems to
demonstrate the contrary. Again, we have yet to find a case where this
approach has be the source of a specific problem.

> i have not spelled it out explicitly, seeking instead to use logical
> reasoning rather than make categorical statements, but this is
> actually a really serious problem.

Again, experience seems to indicate this is not as long as the
definition of the architecture is careful in how the reserved bits get
allocated in the future.


> > IE. Actual examples where
> > this has been cause of breakage in the past ?
> 
> with PowerISA having only a few implementors, even though i am new to
> PowerISA i do not expect there to be many such examples, simply
> because as i said, for he "Embedded" class processors (Motorola's 32
> bit range, and even the Quorl 64 bit range can be said to be
> targetted at the "Embedded" market of Networking and Routing), these
> will not encounter this as a problem (because of the customised
> firmware that they run).
> 
> it is only the *UNIX* Platform where this matters.
> 
> that leaves, realistically, only the IBM POWER range.  and when there
> is only one vendor, interoperability with other vendors is not
> important when there aren't any other vendors!
> 
> however if PowerISA expands (as we know it would like to), and this
> is not changed, i *guarantee* that when more implementors are
> involved there will be problems, here.
> 
> in other words the approach of ignoring reserved bits has only worked
> (for the past 20 years?) because IBM, realistically, is the only
> (non-Embedded) implementor of UNIX Platform processors.

That wasn't always true and still didn't become a problem.

Again, I really fail to see the smoking gun you seem to be implying
there is.

Cheers,
Ben.



_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] system call (sc) LEV "reserved field"

[Luke Kenneth Casson Leighton]

On Thursday, July 23, 2020, Benjamin Herrenschmidt <benh@kernel.crashing.org>
wrote:

> On Thu, 2020-07-23 at 05:16 +0100, Luke Kenneth Casson Leighton wrote:
> >
> > what i have observed (again this is from RISCV), for instructions
> > that can be "safely ignored" are of a "hint" type.
>
> See what Paul said... the idea is that any change to an existing
> instruction that makes use of formerly reserved bits needs to be done
> in such a way that the ignoring the bits is a correct implementation.


this works well if extremely carefully managed, and planned well in advance.

at no time can the new functionality step *up* or otherwise be anything
other than an optimisation.

In practice that often means the new bits are indeed hints,


this is an accidental discovery!  i had not meant them to be considered
"hints" however now that you mention it, it makes a lot of sense.

lwsync would fall into this category.

 but not
> always, see Paul example about mfocrf.


indeed.  although it makes me nervous, that 2/2 cases quoted have known
issue where they did not go as planned.

i am guessing that mtmsr L=0/1 may also have been extended at some point?
that in an earlier version of the spec it was only L=0?



> > i.e. old sync is a "superset" of lwsync?
>
> sync is a memory barrier. lwsync is a lighter weight version of it, so
> yes, sync is a superset of lwsync.


ok, good.  so it is an optimisation.  as a "hint" type it would be "safe"

hints are only safe to add precisely because they are declared in advance -
long before anyone even implements the very first implementation of the
very first core - to have no change in functionality, merely an opportunity
for performance optimisation.


> > bit did not matter, the trap would call the alternative sync
> > instruction, and return from the trap.
>
> The performance impact of this would make it entirely unusable however.


in that example, yes.  most unfortunate.



>
> > i do not know the full story about what FSL is: without knowing the
> > details i suspect that a trap *could* have sorted things out, there,
> > *if* the older processor was required to raise "illegal instruction"
>
> FSL is Freescale.


ahh ok.


>
>

Some variants of their embedded cores did trap, which
> forced the kernel to do some nasty dynamic patching at runtime to get
> rid of lwsync's on these.


a low-grade version of JIT translation, which is one solution.  expensive,
but a solution.

although i have to ask why, for Embedded, they did not just recompile the
source code, customised for that enduser application.

however this also actually illustrates precisely why i mentioned that for
best results, a spec has to have different platform behaviour for Embedded
as completely separate and distinct from UNIX.

the rules for Embedded in RISCV state that traps are *not* needed for
illegal instructions (it is up to an implementor to decide if they wish to
add them).  the Conformance Test Suite does *not* test for illegal
instructions.

this on the basis that Embedded Markets are typically one-off deployment,
where the toolchain and all build source is "snapshotted" at product
release time, used internally and the source code and toolchain almost
never see the light of day.  mainline upgrades are exceptionally rare.

examples include Trinamic Stepper ASICs, which actually hit the shelves
even before RISCV Standardisation was ratified.

the binaries are in ROM.  whatever trap spec incompatibility with "Latest
and Greatest" are utterly irrelevant for them and their customers.


so in that context, i am slightly confused to hear that for Freescale
*Embedded* processors that there is even a binary incompatibility problem
with lwsync *at all*.

if you have time i'd love to hear more, what's the story there?




> > but because the bit is ignored, there is no way that can be done.
>
> But that's fine because "sync" is a strict superset so ignoring the bit
> works just fine.


in this case, yes.



>
> > so whilst on the face of it, lwsync *sounds* like a counterexample,
> > instead (caveat, i do not know what FSL is), it seems more to support
> > the case *for* use of reserved bits raising illegal instructions [on
> > UNIX platforms]
>
> I fail to follow your logic. The trap case is a complete non-starter
> for an instruction meant to be a lightweight memory barrier.


ok given that it is a "hint" type and not quite suited to "reserved" we are
now at cross purposes.

and, also, it is a special case that happens to gave, as one of its
requirements, to be "lightweight" in order to be useful.

The
> transparent "fallback" to a full memory barrier on the contrary works
> fine. There is never any case where "trapping an emulating" is the
> desired behaviour.


indeed... in this very special case that we have established, by it
effectively being a "hint", falls completely outside of what i am concerned
about


>
> >
> > > > if however reserved bits being set cause an exception, the "old"
> > > > processor stands a chance of emulating the new behaviour (in
> > > > software, even if that's slow), giving it a chance of keeping out
> > > of
> > > > landfill for slightly longer.
> > >
> > > Which is why powerpc tends not to "add bits" to instructions unless
> > > ignoring them is a safe fallback.
> >
> > as we can see above, the example that was given  shows that this is a
> > problematic approach.
>
> I fail to understand the "problem" in your example, maybe you can
> rephrase it ?


we've identified that lwsync is a "hint" so can discount it.

i'll need to take some more time to think about it.  also, it requires
explaining the context.

OpenPOWER is, under the auspices of the Foundation, i gather aiming for
much wider adoption whilst also retaining the reputation for performance
and stability, *and* allowing for extensions and customisation under the
stewardship of the Foundation.

with so few members using POWER up to this point, there has been little in
the way of conflict because there is no "pulling in different directions"
and, as Paul pointed out, a lot of care and thought has gone into any
upgrades.

looking for comparisons - examples from the past - is therefore actually
going to be difficult.


> > i have not spelled it out explicitly, seeking instead to use logical
> > reasoning rather than make categorical statements, but this is
> > actually a really serious problem.
>
> Again, experience seems to indicate this is not as long as the
> definition of the architecture is careful in how the reserved bits get
> allocated in the future.


if they are genuinely synonymous with (effectively) a "hint" then there is
absolutely no problem at all... *as long as* the entire area has been
declared fully and well in advance that it is of type "hint".

for example mtmsr L=1 (which i am assuming perhaps incorrectly was added
long after L=0, is that correct? if not let us assume hypothetically that
it was) absolutely in no way falls into the "safe" category because it
damages bits of MSR unexpectedly.  or only updates some of them.

in this (hypothetical?) case, modern software running on an older UNIX
Platform processor that only supported L=1 could *only* sort things out
*if* a trap was taken.

and yet the spec explicitly says, "don't bother", which is *guaranteed* to
cause problems.


SC having LEV > 1 in the future is also a special case because it is
*already* raising a trap.  no extra cost, there.  things can be sorted
out.... in the trap.



> > in other words the approach of ignoring reserved bits has only worked
> > (for the past 20 years?) because IBM, realistically, is the only
> > (non-Embedded) implementor of UNIX Platform processors.
>
> That wasn't always true and still didn't become a problem.
>
> Again, I really fail to see the smoking gun you seem to be implying
> there is.


no gun.

it is that things have changed, now.

where before the specification could be managed by a few vendors, for a few
vendors (managed extremely well) the intention as i understand it is to
expand the reach of OpenPOWER, and allow dozens if not hundreds of
implementors to participate in its ongoing evolution, just as is done with
RISCV.

under such changed circumstance it is well known and well established that
it is *not safe* to have reserved bits *not* throw exceptions on the UNIX
Platform.

bear in mind that we established that lwsync is a "hint", and that hints
*are* safe to add (as long as the available range is reserved fully, in
advance) and do not and should not raise exceptions because they are
performance optimisations.

l.



-- 
---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] system call (sc) LEV "reserved field"

[Benjamin Herrenschmidt]

On Fri, 2020-07-24 at 17:53 +0100, Luke Kenneth Casson Leighton wrote:
> 
> a low-grade version of JIT translation, which is one solution. 
> expensive, but a solution.
> 
> although i have to ask why, for Embedded, they did not just recompile
> the source code, customised for that enduser application.

Because the distinction between "embedded" and "unix" is very blurry
and in many areas obsolete.

One can compile a single image that is meant to run on a wide variety
of system and even the "embedded" world wants that capability. Either
because they end up running some kind of "upstream" OS image, or
because they don't want to maintain completely different SW images for
all their products, etc...

> however this also actually illustrates precisely why i mentioned that
> for best results, a spec has to have different platform behaviour for
> Embedded as completely separate and distinct from UNIX.

This is not really true anymore. For example, do you consider your cell
phone or your TV "embedded" or "unix" ? They are both running Linux
after all...

> the rules for Embedded in RISCV state that traps are *not* needed for
> illegal instructions (it is up to an implementor to decide if they
> wish to add them).  the Conformance Test Suite does *not* test for
> illegal instructions.
> 
> this on the basis that Embedded Markets are typically one-off
> deployment, where the toolchain and all build source is "snapshotted"
> at product release time, used internally and the source code and
> toolchain almost never see the light of day.  mainline upgrades are
> exceptionally rare.

There are quite a few counter examples even in the "embedded" market.
Especially when it comes to storage appliances. Some of these things
can even run CentOS.

> examples include Trinamic Stepper ASICs, which actually hit the
> shelves even before RISCV Standardisation was ratified.
> 
> the binaries are in ROM.  whatever trap spec incompatibility with
> "Latest and Greatest" are utterly irrelevant for them and their
> customers.
> 
> 
> so in that context, i am slightly confused to hear that for Freescale
> *Embedded* processors that there is even a binary incompatibility
> problem with lwsync *at all*.
> 
> if you have time i'd love to hear more, what's the story there?

I forgot the details, but a specific core variant from FSL screwed up
the decode table and would trap on lwsync instead of ignoring the bit.

> > > but because the bit is ignored, there is no way that can be done.
> > 
> > But that's fine because "sync" is a strict superset so ignoring the
> > bit
> > works just fine.
> 
> in this case, yes.
> 
>  
> > > so whilst on the face of it, lwsync *sounds* like a
> > counterexample,
> > > instead (caveat, i do not know what FSL is), it seems more to
> > support
> > > the case *for* use of reserved bits raising illegal instructions
> > [on
> > > UNIX platforms]
> > 
> > I fail to follow your logic. The trap case is a complete non-
> > starter
> > for an instruction meant to be a lightweight memory barrier. 
> 
> ok given that it is a "hint" type and not quite suited to "reserved"
> we are now at cross purposes.
>  
> and, also, it is a special case that happens to gave, as one of its
> requirements, to be "lightweight" in order to be useful.
> 
> > The
> > transparent "fallback" to a full memory barrier on the contrary
> > works
> > fine. There is never any case where "trapping an emulating" is the
> > desired behaviour.
> 
> indeed... in this very special case that we have established, by it
> effectively being a "hint", falls completely outside of what i am
> concerned about

In *most* cases trap + emulation leads to unusable performances though.

Ben.



_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] system call (sc) LEV "reserved field"

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Mon, Jul 27, 2020 at 5:30 AM Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:

> > although i have to ask why, for Embedded, they did not just recompile
> > the source code, customised for that enduser application.
>
> Because the distinction between "embedded" and "unix" is very blurry

benjamin: this is not good!  specifications should *never* be blurry!
i believe you may mean something different, here, which warrants
investigation.  i think i know why, given that the typical markets for
embedded PowerISA cores are specialist areas (aerospace and so on).
Quorl for example is marketed at routers and definitely crosses over,
yet is still termed "embedded".

> and in many areas obsolete.

ok let's look at the v3.1B document, page viii "Compliancy Subsets".

nowhere in there - even in the scalar fixed-point section - am i
seeing mention of what would be expected for "really resource
constrained i.e. *truly* embedded" markets: being able to drop the
(very expensive, gate-wise) logic decoding for catching illegal
instructions, illegal SPR access, and so on.

in SFS the drop in gate count from not needing a FPU is massive: an
FPU typically dwarfs the size of the main core.  once the FPU is
dropped then relatively speaking further savings such as cutting those
needed for illegal instructions becomes signficant (5% saving, 10%
saving) and in mass-volume markets that's absolutely massive.

let us therefore define "resource-constrained embedded markets" as
"truly" embedded, rather than the blurry definition inspired by the
meme that goes by the moniker "IoT", which now includes 8+ watt
raspberry pi 4 devices that run so hot they need fan cooling.

whilst it comes with the "burden" of needing to snapshot and maintain
the full toolchain (which for customisation and custom extensions is a
hard requirement anyway), one of the key areas in which RISC-V has
been successful is the mass-volume *truly* embedded market.

the canonical example is Western Digital, who up until RISC-V had
their own "truly" embedded custom ISA which has been hyper-efficient
for them, for use in SSDs, HDDs and USB Flash devices.  the reason
they wanted to go with RISC-v is to save on maintenance of their own
custom toolchain (bear in mind that with a custom ISA you need not
only a custom toolchain, you need a custom OS and custom applications
as well!  the cost to them of maintaining this would be enormous!)

here, if the binary size is large, this cuts into the usage allocation
of the on-board NAND and on-board RAM (because they don't have
separate NAND from the customer-allocated area, or separate RAM for
the cacheing of customer's data).

if the NAND and RAM allocation are reduced, that has a very real
detrimental impact on their sales and profitability, especially in
such a highly competitive market!

remember that for WD we are talking sales of billions of units, here.
"a few gates" multiplied by a billion sales can make the difference
between profit and loss.

so they found when converting to RISC-V from their own internal ISA
that binary size increased by (iirc) 20%.  they absolutely had to do
something about this and so set about analysing static instruction
allocation and (i am projecting here as this was 2016) they would by
now likely have used that research to create custom instruction
modifications similar to the VLE Book, except targetted specifically
at their use-case.

none of these custom ISA modifications which result in code-size
reduction that is important to their profitability in a very real way
afaik are made public (i have not checked or heard any news so this
could be wrong).

this was all based on the initial base of *not* having to have the
gates for illegal instruction catching etc., they had to go "custom
firmware, custom toolchain" anyway, and the RISC-V initiative provided
them with a huge cost saving on the toolchain and much more.

so they got to have their cake and eat it:

* cost savings on toolchain, kernel, os and bootloader maintenance (by
being able to ride off the back of RISC-V "official")
* cost savings from the "Embedded RISC-V Platform" already having made
decisions that significantly reduced gate count
* cost savings from further reductions by adding custom instructions
and customising the toolchain to match them.

*this* is *truly* embedded.



> One can compile a single image that is meant to run on a wide variety
> of system and even the "embedded" world wants that capability.

if we are talking about the existing (core) OpenPOWER Foundation
members subset embedded markets, served by their current product
offerings: yes.

if we are talking about the *full* (world-wide) definition of
"embedded" which includes "truly" embedded: absolutely not.

and i have to point out that if OpenPOWER's direction only takes into
consideration the former, it is *guaranteed* that Power will never
extend or see wide adoption into the latter.  the cost-benefit
analysis comes up so short that no mass-volume product manufacturer
will consider it, and quite rightly so, as things stand.

this then is the challenge for OpenPOWER if it wishes to have a wider
reach: to adapt to beyond the needs of the current members, whilst
also - very importantly - respecting the long-standing relationship,
contribution and needs *of* those current members at the same time.

it's a delicate balance to achieve.


> Either
> because they end up running some kind of "upstream" OS image, or
> because they don't want to maintain completely different SW images for
> all their products, etc...

indeed.  and this is perfectly reasonable, for what we've colloquially
termed the "blurry" embedded markets.  the cost savings of not having
to recompile or maintain custom packages etc. - these are enormous
savings, absolutely worthwhile pursuing.

unfortunately, if that expectation then propagates throughout the
entire OpenPOWER community as a "hard expectation" (even to the SFS
Compliancy subset), it *automatically and inherently* excludes any
possibility for  "truly" embedded vendors to consider using POWER,
because they are *prohibited* by the Compliancy Requirements from
dropping the gates that would make the product profitable!


> > however this also actually illustrates precisely why i mentioned that
> > for best results, a spec has to have different platform behaviour for
> > Embedded as completely separate and distinct from UNIX.
>
> This is not really true anymore.

thanks to the meme "IoT" and so on the word "embedded" when in general
circulation has become meaningless, yes.  i am not using the term
"Embedded" in the meaningless sense, i am using it in the original
sense used to describe 8 and 16 bit processor markets (when uprated to
32 and sometimes 64 bit).

Embedded in the sense of Arduino PICs, ATMEL ATSAM3 series, ST Micro
STM32F series, and so on. ARM Cortex M0, M3 style and such.

not the "take the latest 64 bit high performance 2.5 ghz quad-core 8+
watt processor, slap it into a SBC form-factor and call it quotes
embedded quotes" definition of Embedded.

> For example, do you consider your cell
> phone or your TV "embedded" or "unix" ?

UNIX, without a shadow of doubt.  Android is a UNIX Platform.  the
Android kernel *is* the linux kernel, and, if end-users are prepared
to put in some effort, all devices running Android - if there isn't
Treacherous DRM built-in to the boot sequence - can have their OS
entirely replaced by any GNU/Linux distro that's compatible with the
processor.

so no - those are *definitely* UNIX platform devices.

> > this on the basis that Embedded Markets are typically one-off
> > deployment, where the toolchain and all build source is "snapshotted"
> > at product release time, used internally and the source code and
> > toolchain almost never see the light of day.  mainline upgrades are
> > exceptionally rare.
>
> There are quite a few counter examples even in the "embedded" market.
> Especially when it comes to storage appliances. Some of these things
> can even run CentOS.

cool!

however, again: are we referring to "general morphed (blurry)"
definition of Embedded, or "truly" embedded?

> > so in that context, i am slightly confused to hear that for Freescale
> > *Embedded* processors that there is even a binary incompatibility
> > problem with lwsync *at all*.
> >
> > if you have time i'd love to hear more, what's the story there?
>
> I forgot the details, but a specific core variant from FSL screwed up
> the decode table and would trap on lwsync instead of ignoring the bit.

and, given the expectation to have binary interoperability even across
"general" (aka colloquially blurry) Embedded, that would matter (a
lot).

> > indeed... in this very special case that we have established, by it
> > effectively being a "hint", falls completely outside of what i am
> > concerned about
>
> In *most* cases trap + emulation leads to unusable performances though.

this is another sub-topic entirely.  the question one needs to ask is:
*why* is performance unusable?  *why* is the cost so high?

RISC-V (again.  there are many good bits to RISC-V) provides
hardware-level partial-decode of instruction sub-fields.  RA, RB, RS,
RT (or, RISC-V equivalents) and other sub-fields, these are all
available via individual SPRs (CSRs in RISC-V terminology) in the
"illegal instruction" trap, saving huge numbers of mask/shift
operations that make trap-and-emulate much faster and require far less
instructions.

without such hardware-level assistance yes i can see that trap and
emulate would be considered completely unacceptable.

we've diverged quite a lot from the original topic :)  but also i
believe provided some insights into the very different needs of
different markets, which pull in completely polar opposite directions
on the same ISA.

the summary is: the Power Spec is a valued and valuable reflection of
the needs of its community.  the question becomes: does that community
want to see OpenPOWER extend further?  if so, it needs to learn from
the analysis done by the RISC-V founders, who analysed *30 years* of
RISC processor architectures and amalgamated the best bits that they
thought would make a modern ISA *and eco-system* supporting a hugely
diverse range of markets, from tiny ("truly") embedded all the way up
to supercomputers.

l.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] [OpenPOWER-HDL-Cores] system call (sc) LEV "reserved field"

[Hugh Blemings]

Hiya,

Apologies for the top post but I can offer a historical perspective on 
at least part of this I think.

Before I do though a disclaimer - these are my opinions and 
recollections only, certainly not policy! :)

While it may change over time, the focus for OpenPOWER in the embedded 
space with the Open ISA has been firmly in the mid to high end and 
therefore mostly 64 bit space.

So to borrow some of the terminology/examples below, a SoC for a 
Raspberry Pi style device based on OpenPOWER ?  Sure, definitely an area 
of interest.

Getting down into smaller devices - really resource constrained and/or 
native 32 bit only - "truly embedded" - that's an area well served by 
RISC-V and not, I think somewhere we're going to see focus for OpenPOWER 
at present.

Not to say it can't in the future, but I think there is a bias towards 
working in areas the ecosystem as it currently stands knows well - 
playing to current strengths. :)

That said I may still do an OpenPOWER based NTP enabled 7-segment 
display bedside clock one day, but that will be born of stubbornness, 
not because I actually need a 64 bit CPU to do it ;)

Cheers,
Hugh



On 27/7/20 6:00 pm, Luke Kenneth Casson Leighton wrote:
> ---
> crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68
>
> On Mon, Jul 27, 2020 at 5:30 AM Benjamin Herrenschmidt
> <benh@kernel.crashing.org> wrote:
>
>>> although i have to ask why, for Embedded, they did not just recompile
>>> the source code, customised for that enduser application.
>> Because the distinction between "embedded" and "unix" is very blurry
> benjamin: this is not good!  specifications should *never* be blurry!
> i believe you may mean something different, here, which warrants
> investigation.  i think i know why, given that the typical markets for
> embedded PowerISA cores are specialist areas (aerospace and so on).
> Quorl for example is marketed at routers and definitely crosses over,
> yet is still termed "embedded".
>
>> and in many areas obsolete.
> ok let's look at the v3.1B document, page viii "Compliancy Subsets".
>
> nowhere in there - even in the scalar fixed-point section - am i
> seeing mention of what would be expected for "really resource
> constrained i.e. *truly* embedded" markets: being able to drop the
> (very expensive, gate-wise) logic decoding for catching illegal
> instructions, illegal SPR access, and so on.
>
> in SFS the drop in gate count from not needing a FPU is massive: an
> FPU typically dwarfs the size of the main core.  once the FPU is
> dropped then relatively speaking further savings such as cutting those
> needed for illegal instructions becomes signficant (5% saving, 10%
> saving) and in mass-volume markets that's absolutely massive.
>
> let us therefore define "resource-constrained embedded markets" as
> "truly" embedded, rather than the blurry definition inspired by the
> meme that goes by the moniker "IoT", which now includes 8+ watt
> raspberry pi 4 devices that run so hot they need fan cooling.
>
> whilst it comes with the "burden" of needing to snapshot and maintain
> the full toolchain (which for customisation and custom extensions is a
> hard requirement anyway), one of the key areas in which RISC-V has
> been successful is the mass-volume *truly* embedded market.
>
> the canonical example is Western Digital, who up until RISC-V had
> their own "truly" embedded custom ISA which has been hyper-efficient
> for them, for use in SSDs, HDDs and USB Flash devices.  the reason
> they wanted to go with RISC-v is to save on maintenance of their own
> custom toolchain (bear in mind that with a custom ISA you need not
> only a custom toolchain, you need a custom OS and custom applications
> as well!  the cost to them of maintaining this would be enormous!)
>
> here, if the binary size is large, this cuts into the usage allocation
> of the on-board NAND and on-board RAM (because they don't have
> separate NAND from the customer-allocated area, or separate RAM for
> the cacheing of customer's data).
>
> if the NAND and RAM allocation are reduced, that has a very real
> detrimental impact on their sales and profitability, especially in
> such a highly competitive market!
>
> remember that for WD we are talking sales of billions of units, here.
> "a few gates" multiplied by a billion sales can make the difference
> between profit and loss.
>
> so they found when converting to RISC-V from their own internal ISA
> that binary size increased by (iirc) 20%.  they absolutely had to do
> something about this and so set about analysing static instruction
> allocation and (i am projecting here as this was 2016) they would by
> now likely have used that research to create custom instruction
> modifications similar to the VLE Book, except targetted specifically
> at their use-case.
>
> none of these custom ISA modifications which result in code-size
> reduction that is important to their profitability in a very real way
> afaik are made public (i have not checked or heard any news so this
> could be wrong).
>
> this was all based on the initial base of *not* having to have the
> gates for illegal instruction catching etc., they had to go "custom
> firmware, custom toolchain" anyway, and the RISC-V initiative provided
> them with a huge cost saving on the toolchain and much more.
>
> so they got to have their cake and eat it:
>
> * cost savings on toolchain, kernel, os and bootloader maintenance (by
> being able to ride off the back of RISC-V "official")
> * cost savings from the "Embedded RISC-V Platform" already having made
> decisions that significantly reduced gate count
> * cost savings from further reductions by adding custom instructions
> and customising the toolchain to match them.
>
> *this* is *truly* embedded.
>
>
>
>> One can compile a single image that is meant to run on a wide variety
>> of system and even the "embedded" world wants that capability.
> if we are talking about the existing (core) OpenPOWER Foundation
> members subset embedded markets, served by their current product
> offerings: yes.
>
> if we are talking about the *full* (world-wide) definition of
> "embedded" which includes "truly" embedded: absolutely not.
>
> and i have to point out that if OpenPOWER's direction only takes into
> consideration the former, it is *guaranteed* that Power will never
> extend or see wide adoption into the latter.  the cost-benefit
> analysis comes up so short that no mass-volume product manufacturer
> will consider it, and quite rightly so, as things stand.
>
> this then is the challenge for OpenPOWER if it wishes to have a wider
> reach: to adapt to beyond the needs of the current members, whilst
> also - very importantly - respecting the long-standing relationship,
> contribution and needs *of* those current members at the same time.
>
> it's a delicate balance to achieve.
>
>
>> Either
>> because they end up running some kind of "upstream" OS image, or
>> because they don't want to maintain completely different SW images for
>> all their products, etc...
> indeed.  and this is perfectly reasonable, for what we've colloquially
> termed the "blurry" embedded markets.  the cost savings of not having
> to recompile or maintain custom packages etc. - these are enormous
> savings, absolutely worthwhile pursuing.
>
> unfortunately, if that expectation then propagates throughout the
> entire OpenPOWER community as a "hard expectation" (even to the SFS
> Compliancy subset), it *automatically and inherently* excludes any
> possibility for  "truly" embedded vendors to consider using POWER,
> because they are *prohibited* by the Compliancy Requirements from
> dropping the gates that would make the product profitable!
>
>
>>> however this also actually illustrates precisely why i mentioned that
>>> for best results, a spec has to have different platform behaviour for
>>> Embedded as completely separate and distinct from UNIX.
>> This is not really true anymore.
> thanks to the meme "IoT" and so on the word "embedded" when in general
> circulation has become meaningless, yes.  i am not using the term
> "Embedded" in the meaningless sense, i am using it in the original
> sense used to describe 8 and 16 bit processor markets (when uprated to
> 32 and sometimes 64 bit).
>
> Embedded in the sense of Arduino PICs, ATMEL ATSAM3 series, ST Micro
> STM32F series, and so on. ARM Cortex M0, M3 style and such.
>
> not the "take the latest 64 bit high performance 2.5 ghz quad-core 8+
> watt processor, slap it into a SBC form-factor and call it quotes
> embedded quotes" definition of Embedded.
>
>> For example, do you consider your cell
>> phone or your TV "embedded" or "unix" ?
> UNIX, without a shadow of doubt.  Android is a UNIX Platform.  the
> Android kernel *is* the linux kernel, and, if end-users are prepared
> to put in some effort, all devices running Android - if there isn't
> Treacherous DRM built-in to the boot sequence - can have their OS
> entirely replaced by any GNU/Linux distro that's compatible with the
> processor.
>
> so no - those are *definitely* UNIX platform devices.
>
>>> this on the basis that Embedded Markets are typically one-off
>>> deployment, where the toolchain and all build source is "snapshotted"
>>> at product release time, used internally and the source code and
>>> toolchain almost never see the light of day.  mainline upgrades are
>>> exceptionally rare.
>> There are quite a few counter examples even in the "embedded" market.
>> Especially when it comes to storage appliances. Some of these things
>> can even run CentOS.
> cool!
>
> however, again: are we referring to "general morphed (blurry)"
> definition of Embedded, or "truly" embedded?
>
>>> so in that context, i am slightly confused to hear that for Freescale
>>> *Embedded* processors that there is even a binary incompatibility
>>> problem with lwsync *at all*.
>>>
>>> if you have time i'd love to hear more, what's the story there?
>> I forgot the details, but a specific core variant from FSL screwed up
>> the decode table and would trap on lwsync instead of ignoring the bit.
> and, given the expectation to have binary interoperability even across
> "general" (aka colloquially blurry) Embedded, that would matter (a
> lot).
>
>>> indeed... in this very special case that we have established, by it
>>> effectively being a "hint", falls completely outside of what i am
>>> concerned about
>> In *most* cases trap + emulation leads to unusable performances though.
> this is another sub-topic entirely.  the question one needs to ask is:
> *why* is performance unusable?  *why* is the cost so high?
>
> RISC-V (again.  there are many good bits to RISC-V) provides
> hardware-level partial-decode of instruction sub-fields.  RA, RB, RS,
> RT (or, RISC-V equivalents) and other sub-fields, these are all
> available via individual SPRs (CSRs in RISC-V terminology) in the
> "illegal instruction" trap, saving huge numbers of mask/shift
> operations that make trap-and-emulate much faster and require far less
> instructions.
>
> without such hardware-level assistance yes i can see that trap and
> emulate would be considered completely unacceptable.
>
> we've diverged quite a lot from the original topic :)  but also i
> believe provided some insights into the very different needs of
> different markets, which pull in completely polar opposite directions
> on the same ISA.
>
> the summary is: the Power Spec is a valued and valuable reflection of
> the needs of its community.  the question becomes: does that community
> want to see OpenPOWER extend further?  if so, it needs to learn from
> the analysis done by the RISC-V founders, who analysed *30 years* of
> RISC processor architectures and amalgamated the best bits that they
> thought would make a modern ISA *and eco-system* supporting a hugely
> diverse range of markets, from tiny ("truly") embedded all the way up
> to supercomputers.
>
> l.
> _______________________________________________
> OpenPOWER-HDL-Cores mailing list
> OpenPOWER-HDL-Cores@mailinglist.openpowerfoundation.org
> http://lists.mailinglist.openpowerfoundation.org/mailman/listinfo/openpower-hdl-cores
>
_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev

Re: [libre-riscv-dev] [OpenPOWER-HDL-Cores] system call (sc) LEV "reserved field"

[Luke Kenneth Casson Leighton]

On Mon, Jul 27, 2020 at 10:08 AM Hugh Blemings <hugh@blemings.org> wrote:
>
> Hiya,
>
> Apologies for the top post but I can offer a historical perspective on
> at least part of this I think.

appreciated.

> Before I do though a disclaimer - these are my opinions and
> recollections only, certainly not policy! :)

always important to know where things are coming from.

> While it may change over time, the focus for OpenPOWER in the embedded
> space with the Open ISA has been firmly in the mid to high end and
> therefore mostly 64 bit space.

you may be interested to know that Andes Technology (a company in
Taiwan that has traditionally done DSPs focussing on audio processing:
think e.g. "C-Media" USB Audio devices: massive volume) has become the
de-facto creator and maintainer of RV32 Linux.

this because whilst everyone else is abandoning 32-bit because
640k^H^H^H^H 4GB should be enough for everyone, turns out this isn't
true any more thanks mostly to web browser memory consumption, Andes
Technology recognises that, in the huge market their customers *need*
the space and power saving that comes with a 32 bit ISA.

and it's enormous.

the jump in power consumption suffered by ARM when they went from the
32-bit Cortex A7 (which has a 48-bit address space) to the 64-bit
Cortex A53 was a whopping 10 to *15* percent for the same level of
DMIPS and actual real-world performance.

this because the increase in instruction length that inherently goes
with a 64 bit ISA required vendors using the Cortex A53 to lay down an
extra 50% larger L1 I-Cache.  with L1 caches being a major part of
power consumption, real-world performance/watt figures took a
nose-dive.


> So to borrow some of the terminology/examples below, a SoC for a
> Raspberry Pi style device based on OpenPOWER ?  Sure, definitely an area
> of interest.

cool.

> Getting down into smaller devices - really resource constrained and/or
> native 32 bit only - "truly embedded" - that's an area well served by
> RISC-V and not, I think somewhere we're going to see focus for OpenPOWER
> at present.

the advantage that they had was not being constrained by what is,
slightly unfairly / inaccurately, termed "mistakes of the past".  they
had a clean sheet, basically.

could OpenPOWER follow that?  with careful planning, and given the
care that's already been taken (as Paul mentioned), quite probably
yes.

*should* the OpenPOWER Foundation Members get together and work on
that?  mmm... there would have to be clear benefits and a chance of
market success and adoption that warranted the effort.  and, i think,
here, given that as you say, RISC-V already serves this area well,
it's not clear that it would be worthwhile doing.

i.e. unless it's something that was invested in (multi-year) by
something Foundation-like and University-collaboration-like (with
associated large DARPA/other Grants to back it up, as happened with
RISC-V) you'd need a well-funded OPF stakeholder that was actively
interested in pursuing it, and given the resources involved they'd
more than likely evaluate RISC-V as the better cost-benefit option.


> Not to say it can't in the future, but I think there is a bias towards
> working in areas the ecosystem as it currently stands knows well -
> playing to current strengths. :)

yeh absolutely.  incremental steps, with consensus.

> That said I may still do an OpenPOWER based NTP enabled 7-segment
> display bedside clock one day, but that will be born of stubbornness,
> not because I actually need a 64 bit CPU to do it ;)

:)  with added realistic and annoying analog buzzer alarm (a la
Groundhog Day), to boot.

l.

_______________________________________________
libre-riscv-dev mailing list
libre-riscv-dev@lists.libre-riscv.org
http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev