Add struct functionality

[rkalis]

Structs can be defined by the implementer of a contract like this:

struct PriceMessage {
    bytes4 blockheight;
    bytes4 price;
}

The CashScript SDK can automatically serialise JSON data into a byte array containing these concatenated struct members. This serialised struct can then be passed into a contract function and the compiled script can use OP_SPLIT and OP_BIN2NUM to extract the correct struct members where needed.

To make sure the components can be extracted, they should use sized bytes types, so the usefulness of these structs is dependent on issue #20.

[rkalis]

You could probably have destructured assignment like this:

{ bytes4 blockheight, bytes4 price } = message;

Or direct access llike this:

message.price

[mr-zwets]

this is no longer only handy for datamessages used with checkDataSig but also for nft state kept by contracts, or to defined the commitments the contract interacts with!

for example:

struct LoanCommitment{
    bytes8 blockheight;
    bytes20 loanerPubKeyHash;
    bytes8 borrowedTokenAmount;
}

inspired by https://bitcoincashresearch.org/t/stable-asset-token-incentivized-floating-peg/1206

the idea that you could destructured borrowedTokenAmount is quite powerful

[mr-zwets]

Some remarks on Structs:

The byte length of all member elements except the last one should include the length with bytesX to be able to access members with .split(), for the last item it's an open decision whether to allow for pre-defined length or not.

There's 2 scenarios

1. a bytes item is passed in the function arguments and should be cast as the struct type
2. a user defines a struct in the contract logic

casting as struct

LoanCommitment commitment = tx.inputs[0].nftCommitment

user defined structs:

LoanCommitment commitment = LoanCommitment({
    blockheight: 0x1234567890ABCDEF,
    loanerPubKeyHash: 0xabcdefabcdefabcdefabcdefabcdefabcdefabcdef,
    borrowedTokenAmount: 0x1122334455667788
});

So it seems for both scenarios we would allow for user-defined types, something CashScript doesn't have currently.

We also need to think about whether is always safe for users to cast a struct (what if the bytestring doesn't match the expected length). If the struct has a fixed length (so all member items have a fixed length) it would be possible for users to do checks like this

bytes commitmentBytes = tx.inputs[0].nftCommitment
require(commitment == LoanCommitment.length)
LoanCommitment commitment = LoanCommitment(commitmentBytes)

[SayoshiNakamario]

A lot of the nuance goes over my head, but the ability to define a struct then refer to it in the cashscript logic would be nice rather than having to manually do all the split()'s we currently do. I'm not sure the benefits of passing a bytes item to a function over defining it in the contract - the latter seems easier to understand since everything is contained in the same spot, and is more familiar from a Solidity mindset.

struct LoanCommitment ({
    bytes6 blockheight,
    bytes20 loanerPubKeyHash,
    bytes6 borrowedTokenAmount
});

LoanCommitment loan = tx.inputs[0].nftCommitment

require(tx.inputs[1].lockingBytecode == new LockingBytecodeP2PKH(loan.loanerPubKeyHash))

[mr-zwets]

thanks for your input @SayoshiNakamario !

the ability to define a struct then refer to it in the cashscript logic would be nice rather than having to manually do all the split()'s we currently do

Yeah the .split() for state with more than 2 items quickly becomes involved, and currently even for getting a middle part of a bytestring (although this would be improved with the proposed 'slice' syntax of #170 )

// if you want all three items currently we'd do this
bytes6 blockheight, bytes remainingState = tx.inputs[0].nftCommitment.split(6);
bytes20 loanerPubKeyHash, bytes6 borrowedTokenAmount = remainingState.split(20);

// for one item
bytes20 loanerPubKeyHash = tx.inputs[0].nftCommitment.split(6)[1].split(20)[0];

// for one item with 'slice' syntax
bytes20 loanerPubKeyHash = tx.inputs[0].nftCommitment.slice(6, 20); 

I did learn that solidity does not support Struct de-structuring... I do think only having member syntax (with struct.member) would result in a different compiled output vs declared, reusable variables.

The new syntax with Structs is not really shorter than before, but it does avoid having to think about split() indexes and for easier typing of the last element borrowedTokenAmount

// before
bytes6 blockheight, bytes remainingState = tx.inputs[0].nftCommitment.split(6);
bytes20 loanerPubKeyHash, bytes borrowedTokenAmount = remainingState.split(20);

// after
LoanCommitment commitment= tx.inputs[0].nftCommitment;
{bytes6 blockheight, bytes20 loanerPubKeyHash, bytes6 borrowedTokenAmount} = commitment;

Let's also compare creating a bytestring

// before
bytes32 newCommitment = bytes6(intBlockheight) + loanerPubKeyHash +bytes6(borrowedTokenAmount)

// after
LoanCommitment newCommitment = LoanCommitment({
    blockheight: bytes6(intBlockheight),
    loanerPubKeyHash: loanerPubKeyHash ,
    borrowedTokenAmount: bytes6(borrowedTokenAmount)
});

Clearly using structs here results in more code, but the advantage is the explicit length checks for each of the members of LoanCommitment.

Structs largely replace .split() usage because /split() signals a single bytestring actually has independent members with different meaning.

To me the main benefits of structs is clearly during development when the layout of the Struct is still changing, the type checks and number of member checks flow through your entire contract.

It also standardizes how contract authors should display the structure of their data, by defining it at the top of the file in structs.

Structs increase the verbosity of the language but also increase contract safety considerably.

[mr-zwets]

One common usecase for .split() is the following

bytes32 tokenId= tx.inputs[0].tokenCategory.split(32)[0];

It's interesting to consider how Structs might or might not affect this.

The tricky part here is that the last member is optional.
If we were to support optional members, we could standardized a struct for this:

Struct IntrospectionTokenCategory{
  32 bytes tokenId
  ?byte capability
}

{bytes32 tokenId}= tx.inputs[0].tokenCategory;

[mr-zwets]

One strong argument in favor of struct functionality is that we see people defining the data structure of their nft commitment bytestring as a top-level comment

I used the p_bond_main.cash contract of @A60AB5450353F40E as an example of this but now I realize it's a special example because it mainly uses variable lengths types inside the data-structure:

// hashedParams is defined as byte concatenation of:
// - bytes1(completionBytecode.length())
// - bytes(completionBytecode),
// - bytes1(feePayoutBytecode.length()),
// - bytes(feePayoutBytecode),
// - bytes8(rewardPayoutAmount),
// - bytes1(rewardPayoutBytecode.length()),
// - bytes(rewardPayoutBytecode),
// - bytes1(dexBytecode.length()),
// - bytes(dexBytecode)

This is an interesting example that you can work around the 'fixed length' requirement.

A problem arises however when a variable length member of a struct would be redefined

struct LoanCommitment ({
    bytes6 blockheight,
    bytes loanerLockingBytecode,
    bytes6 borrowedTokenAmount
});

bytes20 exampleBytes20 = 0xabcdefabcdefabcdefabcdefabcdefabcdefabcdef
LoanCommitment commitment = LoanCommitment({
    blockheight: 0x1234567890ABCDEF,
    loanerLockingBytecode: exampleBytes20,
    borrowedTokenAmount: 0x1122334455667788
});
bytes40 exampleBytes40 = 0xabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdef
commitment.loanerLockingBytecode = exampleBytes40 

interesting questions

• should structs be immutable or can you do member re-assignment
• some items like LockingBytecode are inherently variable length with P2PKH, P2SH20 & P2SH32. could variable length members be something worth considering, with the underlying length workaround used above

member reassignment for variable length members would also have to update the member's length.

I think updating members in a struct would be an important as users would read an nft commitment as a struct data type and and then want to mutate this data structure to enforce is as the next nft commitment

[mr-zwets]

also a further consideration is that with the May upgrade we'll have bigint so bytes8(someInt) might not work reliably anymore

when working with math it might no be clear to what length to cast ints, and how much the overhead of this would be, the number might be only a few bytes so the fixed encoding might add a lot of overhead for a struct like

struct exampleStruct {
  int someSmallOrBigInt0
  int someSmallOrBigInt1
  int someSmallOrBigInt2
}

the compiler would turn that into 6 items, for example:
  length= 2 bytes, bytes someSmallOrBigInt0
  length= 10 bytes, bytes someSmallOrBigInt1
  length= 3 bytes, bytes someSmallOrBigInt2

The hard part becomes that to get the last member the compiler will have to get the variable lengths of all the previous items

// to get someSmallOrBigInt2 the compiler would have to do the following
bytes commitment= tx.inputs[0].nftCommitment;
bytes lengthItem0, bytes remaining0 = commitment.split(1)[0]
bytes lengthItem1, bytes remaining1 = remaining0.split(lengthItem0)[0]
bytes someSmallOrBigInt2 = remaining1.split(lengthItem1)[1]

the advantage is that it would allow all types in structs including int and string 🤔

We can start with the simple struct type, but there is definitely a rationale to allow for variable size structs.
These two types of structs could be called differently to signify the compiler treats them differently (variable length vs fixed length)
byteStruct & struct where struct would allow any type similar to other programming languages 👍

[mr-zwets]

multi-line de-structuring

With struct de-structuring we would see much more multi-line assignments like this:

      {
        bytes2 item1
        bytes2 item2
        bytes6 item3
        byte6 item4
        byte6 item5
        byte4 item6
      } = structData;

We might want to support the constant keyword in front.

Thinking about it, I'm not even sure we want to allow for re-assigning these de-structured variables.
If we do, we might want a readonly keyword for structs that signals member fields cannot be reassigned...

Constructing a new commitment

Now if item5 and item6 in the struct are updated to construct new commitment data

// before
bytes26 newCommitment = structData.split(16)[0] + newItem5 + newItem6;

// new method, mutating the original `structData`
structData.item5 = newItem5
structData.item6 = newItem6

// new method, copying, then mutating the original `structData`
newStruct = structData
newStruct.item5 = newItem5
newStruct.item6 = newItem6

// or alternatively
newStruct = StructType({
    item1: structData.item1,  // Copy unchanged fields
    item2: structData.item2,
    item3: structData.item3,
    item4: structData.item4,
    item5: newItem5,  // Modify specific fields
    item6: newItem6
});

It could be hard for the compiler to match the efficiency of the original .split() syntax in this case.

Additional questions

A practical question becomes, do we still allow .split() on a struct? or is the bytestring as a whole not accessible this way anymore?
if we do allow structData.split(x) what about the other bytes operators and members?

      structData.length()
      structData.reverse()

What about casting from a struct type back to bytes ?

[mr-zwets]

My current idea is that we can do a development roadmap in phases which can be rolled out as individual improvements:

1. implement fixed-length bytesStruct types, only usable through type-casts
2. allow a variable length last item in bytesStruct (this would disallow struct.length to be used for it)
3. allow for struct assignments and mutating member fields
4. bytesStruct as constructor/function params (including support in the SDK)

somewhere in the further future, allow for a general struct type with variable length member fields

The main rationale for this ordering is that it will allow bytesStruct types to be immediately useful for developer parsing the result of .nftCommitment

With phase 2 all usecases we commonly see for .nftCommitment would be support
There still would not be good support for assembling a new nft commitment state, this would still rely on .split()

then in phase 3 this would be solved with proper struct assignment and mutation

then lastly in phase4 usecases like the original AnyHedge would be fully supported where the bytes message for a dataSignature is passed as a function argument. However they would be able to explicitly cast the bytestring to the struct type since phase 1.