[Proposal] Forcing the sequencer to actually submit data to L1

In Aztec Connect, users had to trust that the Sequencer would actually submit encrypted note preimages to L1, as calldata. There was no enforcement at the protocol level, to ensure that every note commitment was accompanied by its corresponding encrypted note preimage.

We need to enforce this, when we build Aztec. Otherwise, an app’s data might never get broadcast!

The Problem:

Suppose a user has a note_preimage, and they want a guarantee that the encrypted_note_preimage = enc(note_preimage) will be definitely be submitted when the note_hash is submitted.

Note: we don’t care about how enc() works, in this post.

The Public Inputs ABI for an Aztec function cannot be dynamic in size. It is a fixed-size struct. But we want to allow users to be able to design custom notes, which might have custom data sizes. Therefore, the encrypted_note_preimage might be variable in size, depending on the app.

Proposal:

tl;dr
Expose, a new field log_hash as a public input, which can be the hash of any data. Provide a way for that data to be submitted on-chain, re-compute the hash on-chain, and compare it against the public input log_hash. Reject the rollup block if this check fails.

Essentially, we allow a single field element – compressed_log – to represent “all of the data I wish to submit as an event”. Naturally, this single field element will be a sha-256 hash of the unpacked event data.

I’m ignoring EIP-4844 here. I’ll talk about it in the current world, where calldata is usually sha-256-hashed in order to be validated. It’s easy ™ to migrate these concepts to EIP-4844, eventually.

We’ll add four new fields to the Public Inputs ABI of an Aztec function.

struct PrivateCircuitPublicInputs = {
    encryped_log_hash: Field,
    unencrypted_log_hash: Field,
    encryped_log_preimage_length: u32,
    unencrypted_log_preimage_length: u32,
}
  • encryped_log_hash: Field: is a single field, which MUST be the sha256-hash of the encrypted data we wish to submit via L1.
  • unencrypted_log_hash: Field: is a single field, which MUST be the sha256-hash of the unencrypted data we wish to submit via L1.
  • encryped_log_preimage_length: the size of the preimage data, so that the gas cost of this request can be measured by circuits, without actually needing to feed in the variable-length data.
  • unencrypted_log_preimage_length: the size of the preimage data, so that the gas cost of this request can be measured by circuits.

Note: I’ve kept encryted and unencrypted data separate, for now. My thinking being: it’ll be easier this way for an aztecRPCServer to identify which data it should attempt to decrypt. Otherwise, it could waste effort. Maybe there’s a nice encoding which would allow the preimages to be combined, whilst also distinguishing which ‘substrings’ of the blob are encrypted vs unencrypted.

When a user submits their Private Kernel Proof to the tx pool, they will also need to provide the xxx_log_preimages in that submission.

For every tx which the sequencer includes in their L2 block, they will need to submit the xxx_log_preimages to L1.

TBD: this data can either be submitted as part of the ‘main’ rollup tx; or submitted in advance to a separate function, which would hash the data and store the hash for later querying by the ‘main’ rollup.

The xxx_log_hash and xxx_log_preimage_length fields are percolated all the way up to L1 (unless xxx_log_hash == 0, in which case we can maybe optimise it away). The xxx_log_preimage data blob is also submitted to L1, where it is sha256-hashed.

When the sequencer submits this L2 block to L1, the Rollup Contract will be able to decode the calldata to identify each individual xxx_log_hash field of each individual tx. It can then compare this xxx_log_hash value against the sha256(xxx_log_preimage):

require(encryped_log_preimage_length == encrypted_log_preimage.length);
require(encryped_log_hash == sha256(encryped_log_preimage);

require(unencrypted_log_preimage_length == unencrypted_log_preimage.length);
require(unencrypted_log_hash == sha256(unencrypted_log_preimage);

Why call it “log”?

Why not name it something specifically about “note encryption”?

Because this field can now be used more generally than just for note encryption!

Any event data can be emitted with this approach!

  • Custom event information (encrypted or publicly visible)
  • Custom note preimages
  • Custom nullifier preimages (see Vitalik’s nice blog for a nice example of why an encrypted nullifier preimage might be useful for apps!).

Note: This does not enforce that the encrypted data is correct. That’s another problem, for another thread. Even for unencrypted blobs of event data, this does not enforce that the data is used by the app circuit in any way. It’s up to the app to ensure this (or to not bother ensuring this) via their Noir Contract logic!
All this proposal ensures is that the Sequencer cannot submit a valid rollup unless they also make all requested event data available on L1.


Continuing the discussion further

The above is the meat of this proposal, and is hopefully not too controversial.

The following tries to standardise some encodings of ‘event data’, to make it easier for an aztecRPCServer to process that data.

It’s important to note that the below suggestions might be over-fitted to “brute force” syncing, where a user’s aztecRPCServer must trial-decrypt every log.
We’re hoping to move away from that, eventually, but for now here are some ideas.

Standardising the compressed_log_preimage data encoding:

Note:
If there’s an encoding (not to be confused with ‘encryption’!) which allows encrypted and unencrypted data to be smooshed together in one blob, that would be nice.

Regardless, we can devise an abi-encoding the data blobs, so that an aztecRPCServer can distinguish between different actual events.

Here’s a suggestion:

  • header = contract_address
  • body = encoding of log_preimage
  • Data submitted to L1: enc_1(header) || enc_2(body)

The idea here would be: the header can be trial-decrypted (at least, in our initial protocol) using a user’s global enc/decryption key. It would also be much faster to only trial-decrypt one field element, rather than the whole blob, because the user will be trial-decrypting millions of times (and so parallelising the decryption of the whole of every message would only waste cores that could otherwise be trial-decrypting the next headers).

Then once they have a match for the header, we need to decrypt the body.

Option 1

  • Use the same decryption key as was used for the header. I.e. one decryption key for all contracts.

Option 2

  • Use a different decryption key per contract. Something like BIP-32 can be used to allow users to derive each-others’ encryption/decryption keys on a per-contract basis.

Option 3

  • Allow Noir Contract developers to write custom encryption/decryption logic for how the body of their contract’s encrypted logs can be decrypted. If a function_selector for the Noir Contract’s custom decryption function is included in the header, the aztecRPCServer could pass the body of each successfully-decrypted header to the appropriate decryption function.

Regardless of which option we choose, I think we will need to include in the body the function_selector(s) of the Noir Contract functions which can serialise custom notes and compute custom nullifiers.


Rolling with Option 3, for example’s sake:

  • header = contract_address || body_decryption_function_selector

  • Here’s a suggestion for the encoding of the body:
    number_of_events || first_event_processing_function_selector || first_event_length || first_event_data || etc...

The body_decryption_function_selector is just a suggestion to allow custom decryption algorithms to be written in Noir Contracts. To be discussed more.

The first_event_processing_function_selector is some function selector which can ‘process’ this log data. It might be a note serialisation (and storage) function. It might compute the nullifier of a note. It might process some app-specific data.

Note: standardising an encoding is really only for the benefit of the aztecRPCServer software, so that it can process blob after blob of event data. Maybe there’s a better encoding. Maybe various standards of encodings will be developed over time by the community. Noir Contracts themselves could dictate the encoding and be fed the opaque blobs to decrypt.

Example

Emitting data required for a simple transfer, but not performing the encryption calculation within the circuit.

Suppose, outside of the circuit, the following data has been encoded, and then encrypted:

  • header:
    • contract_address: 0x...
    • body_decryption_function_selector (suggestion under option 3 above)
  • body:
    • Number of events coming from the transfer function: 1
    • first_event_processing_function_selector: process_new_note_preimage.function_selector
    • Length of the first event: 4 * Field.length
    • First event arg (sender): 0x...
    • Second event arg (receiver): 0x...
    • Third event arg (value): 10
    • Fourth event arg (randomness): 0x...

Noir Pseudocode:

/// File: zk_token.nr

use aztec::events::emit_encrypted_log_hash;
use aztec::notes::ValueNote;
use aztec::notes::ValueNoteLength;
// other imports not shown

contract ExampleToken {
    balances: Map<Field, Set<Note>>;
    
    // Note:
    // In this example, we won't go down the rabbit hole
    // of _proving_ correct encryption, since we want to
    // focus on the topic at hand.
    // So we just pass-in the sha256 hash of some encrypted
    // data blob. The circuit doesn't isn't checking it's
    // correct, for the sake of keeping the example brief!
    //
    fn transfer(
        amount: Field,
        sender: Field,
        recipient: Field,
        hash_of_encrypted_recipient_note: u256
        preimage_length: u32
    ) {

        // Not shown:
        //   - Do the logic of a transfer, as we've seen many times.

        emit_encrypted_log_hash(
            hash_of_encrypted_recipient_note,
            preimage_length
        );
    }
    
    // Provides an example of an unconstrained function which
    // could process a note for the aztecRPCServer,
    // provided its function_selector is included in the
    // encrypted log!
    unconstrained fn process_new_note_preimage(
        note_data: [Fields, ValueNoteLength]
    ) {
        let contract_address = note_data[0];
        let storage_slot = note_data[1];
        let note_fields = note_data.slice(2);
        
        // The oracle will keep notes in a special db collection.
        oracle.store_as_note(
            contract_address,
            storage_slot,
            note_fields
        );
    }
}


/// File: events.nr

// Provides a way to emit an opaque hash value, without 
// actually validating or computing that hash value from
// any underlying data.
// Useful for applications where proving correct encryption
// isn't needed, and all we need is to ensure data availability
// on L1.
fn emit_encrypted_log_hash(
    encrypted_log_hash,
    encrypted_log_preimage_length
) {
    // Ensure it hasn't already been pushed-to.
    assert(context.public_inputs.encrypted_log_hash == 0);
    
    context.public_inputs.encrypted_log_hash = encrypted_log_hash;
    context.public_inputs.encrypted_log_preimage_length = encrypted_log_preimage_length;
}


global encrypted_log_preimage: [Field]; // Would be cool, if possible,
                                        // to derive its length at 
                                        // compile-time.

// Not used in this example, but interesting to see:

// Pushes more event data to an in-memory array.
// 
// Once all events have been pushed (at the end
// of the `main` function), we can compute the 
// hash through a separate function.
fn push_encrypted_log_data<N>(
    encrypted_log_data: [Field; N]
) {
    // E.g. push the length of the next event first:
    encrypted_log_preimage.push(N);
    // Then push the actual data:
    encrypted_log_preimage.push_array(encrypted_log_data);
}

// Hash whatever event data is in memory:
// Pseudocode: because there's som tricky 'length' concepts here.
fn compute_and_emit_encrypted_log_hash() {
    log_hash: Field = pedersen(encrypted_log_preimage);
    
    emit_encrypted_log_hash(log_hash, log_hash.length());
}

Edit: significant edits, following Santiago’s message.
Edit: rename: aztec-node → aztecRPCServer
Edit: rename: compressed_xxx_log to xxx_log_hash

88 Likes

There’s a painful detail that I forgot to mention:

The encrypted_log_hash is a Field type, in this proposal, but the output of a sha256 is 256-bits.

Either the Field type should be changed to be a u256 type (a tuple of Field elements), or the output of the sha256 function should be hashed/truncated to a Field element.

84 Likes

I’m a bit confused by this example: the process_new_note_preimage seems to indicate what to do with the note, but not how to decrypt it (it seems like it’s receiving an already decrypted note). Is the idea to allow custom decryption or custom processing?


Aside, let’s say we don’t want to enable custom processing or encryption. What “type” of things should we be considering?

  • Public events (same as Ethereum)
  • Encrypted events (“hey, you got an erc20 transfer!”)
  • Encrypted note preimages (we need to send the recipient the note encrypted with their pub key, the recipient user’s aztec node will store these in the local db)
  • Encrypted nullifier preimages (do we need to emit these? could we just emit a plain event with the extra data, along with the vanilla nullifier?)

Also, for encrypted data, do we want to support data visible by more than one user?

72 Likes

I’m a bit confused by this example

Thanks - I confused two concepts in my example, so it was nonsense. I’ve gone back and made significant edits to the latter half of the proposal, making a clear distinction between custom decryption functions (which are a tentative proposal that might not be needed) and custom log processing functions (which I think are absolutely needed, in order to process custom note and nullifier data).

What “type” of things should we be considering?

I think we need to support the emission of all of the things you list. I think submitting them all via the proposed log_hash field is nice and tidy. But it would need custom log processing functions, to work nicely.

Also, for encrypted data, do we want to support data visible by more than one user?

Data can be encrypted in a way where multiple parties are able to decrypt it. So I think a single log of data would still work in such cases.

68 Likes

to make it easier for an aztec-node to process that data.

References to aztec-node in the article should be Aztec RPC Server, where encryption and decryption happen.

When a user submits their Private Kernel Proof to the tx pool, they will also need to provide the xxx_log_preimages in that submission.

iiuc, 1 tx has 1 final xxx_log_preimage, and it has to conform to a encoding similar to log body:
number_of_logs || first_log_length || first_log || second_log_length...

Each log is for a different account: enc_1(header, account_A) || enc_2(body, account_A).

The first_event_processing_function_selector is some function selector which can ‘process’ this log data.

We might not need to include the function selector for processing data. An Aztec RPC Server doesn’t do anything with the decrypted data directly. It just saves the decrypted data along with the contract address and an identifier.

If a user wants to know anything, they query it via the contract. Functions in that contract then call the oracle (in Aztec RPC Server) to get the data, and the functions will know how to interpret the decrypted data they receive from the oracle, process it, and return to the user. So instead of a function selector, we might just need to include an identifier for the data: storage slot, event id, etc.

Except for nullifiers, the Aztec RPC Server should know the nullifier when it decrypts a note, so that it can remove the note later on when it sees the nullifier being emitted from a block. Or maybe we can use this new event mechanism to somehow let the server know when to remove a piece of data…

70 Likes

Thanks for the renaming correction - I’ve updated the original post.

Ohhh this is a good point. I think I actually made a mistake in the encoding I suggested in the original post. Yes, a single tx (or even a single function) might need to emit events to several recipients (not just one recipient). So actually the encoding you suggest here is better, because it allows different people to decrypt different events within one logs_preimage

I was hoping to obscure the number_of_logs and the length of each individual log, as a way of providing greater ‘function hiding’, since the number of logs gives away information about the tx which might have been executed. The length of the giant log_preimage leaked some information anyway, but this encoding you suggest leaks a little more, but might be necessary.
Might be worth us thinking about encodings which hide these things.

We might not need to include the function selector for processing data

I’d like to explore this more :slight_smile:

5 Likes

A follow-on thoughts:

  • What should the kernel circuit do?
  • What should the rollup circuit do?

Suppose a tx comprises a few functions fn A → fn B → fn C, and they all emit events.

Then the logs_hash of each function’s Public Inputs ABI will be populated.

Now, does the kernel circuit:

  • a) squash all of the logs_hash values into a single kernel circuit public input field called logs_hash (using sha-256); or
  • b) push each function’s individual logs_hash value into a new array of the kernel circuit’s public inputs?

Option (b) is simpler to implement, but it might restrict the depth of a call stack (since if each function were to call an event, the array would quickly reach max capacity).

Option (a) would allow the amount of logging to only be bounded by L1 calldata + hashing costs. But we’d need to ‘remember’ the ‘wonkiness’ of how the logs were hashed together.

Let’s solve for option (a), because it’s better imo.

The kernel circuit will receive as input:

  • a single logs_hash value, representing the hashing of all logs_hash values of functions which have been processed (by previous kernel iterations) up until this point. Let’s call it prev_kernel_logs_hash.
  • a new function’s call stack item, which contains the public inputs of that function, which includes the logs_hash of that function. Let’s call it new_fn_logs_hash.

The kernel circuit needs to compute new_kernel_logs_hash = sha256(prev_kernel_logs_hash, new_fn_logs_hash).

The aztecRPCServer will need to retain a memory of the ordering in which each functions’ log_preimages were hashed together by the kernel circuit, so that the log_preimages can be encoded correctly for submission to the tx pool, so that the L1 contract can correctly re-calculate the final logs_hash which is submitted to L1.


The base/merge rollup circuits have a similar problem. For the merge rollup circuit in particular, we don’t want to support an array of log_hashes, because the higher up the rollup tree you get, the bigger the array would need to be.
We’ve already solved this, when we designed how contract deployment data should be hashed by the merge rollup circuit, into a single public input.

In the base/merge rollup circuits, we’ll simple take the log hashes of the two input proofs, and sha256-hash them together into a single output field (which forms part of the public inputs of the rollup circuit).

The sequencer will need to retain a memory of how the log preimages have been hashed together, so that they can be encoded correctly for submission to L1, so that L1 can correctly re-compute the sha256 hash.

For clarity: there will be a single logs_hash submitted to L1 representing all of the log data from all txs in the rollup block!.

Note: There is still some thinking to be done if we wanted to allow ‘wonky’ rollup proof trees; we’d need to encode the wonkiness of the tree, so that the hashing of calldata can be performed in the correct ‘wonky’ way on L1.
I’ll dig out the encoding I suggested to Lasse.

5 Likes

We need to encode the log preimages in such a way that the Rollup contract is then able to reconstruct the final hash. To achieve this we need to convey to which kernel and to which kernel iteration/function of the kernel the specific log preimages belong. Given that the data length is variable I think it’s natural to prefix the individual kernel’s blocks and individual preimages with their respective lengths.

This is how encoding of 3 kernels would look like:

|| K1_LEN | P1_LEN | P1 | P2_LEN | P2 | P3_LEN | P3 || || K2_LEN | P1_LEN | P1 || || K3_LEN | P1_LEN | P1 | P2_LEN | P2 | P3_LEN | P3 | P4_LEN | P4 ||

Note: Kernel (K1) contains 3 function invocations, kernel 2 (K2) 1 invocation and kernel 3 (K3) 4 invocations.

Note 2: K?_LEN does not include length of itself (e.g. K2_LEN is the length of | P1_LEN | P1 | )

Then the log hashes would be computed as:

log_hash_K1 = hash(hash(hash(P1), P2), P3)
log_hash_K2 = hash(P1)
log_hash_K3 = hash(hash(hash(hash(P1), P2), P3), P4)

Once this is done the pairs of log hashes would get added to calldata_hash_inputs in base_rollup_circuit (e.g. log_hash_K1 and log_hash_K2 in base rollup circuit 1 etc.) and we are done.

If we’ll keep the encrypted and unencrypted data separate then in base rollup circuit we would concatenate encrypted_log_hash_K1, unencrypted_log_hash_K1, encrypted_log_hash_K2, unencrypted_log_hash_K2 to calldata_hash_inputs.

I would say keeping the encrypted and unencrypted data separate is ok but if we would think it’s worth it to decrease the amount of hashing modifying the encodying should be straightforward.

In the final calldata going on chain I would put the data at the end:

 * ...
 *  | newContractData (each element 52 bytes)
 *  | len(l1ToL2Messages) denoted f
 *  | l1ToL2Messages (each element 32 bytes)
 *  | len(newEncryptedLogs)
 *  | newEncryptedLogs
 *  | len(newUnencryptedLogs)
 *  | newUnencryptedLogs
 *  | ---

Does this sound good to everyone?

2 Likes

After torturing @LHerskind with my insufficiently documented changes in Decoder I realized that we didn’t really describe how the changes are going to be incorporated into the codebase.

The plan is that the logs preimages (or simply logs) will be hashed in Noir and the resulting hashes of individual Noir function calls will be emitted as a part of private circuit public inputs. The hashes of the logs hashes of the individual iterations will then be “recursively” hashed in the kernel circuit. The final/accumulated hash will be part of kernel’s public inputs. Base rollup circuit will then include this final logs hash as a preimage to the calldata hash.

This implies that we will need to reproduce all the hashing of preimages and the kernel’s hashing of logs hashes inside Decoder.sol`.

Including all the logs preimages inside the L2 block calldata will potentially result in the L1 tx to be too big for Ethereum nodes to accept. This could be solved by computing the hash in a separate tx and storing it. This however is not relevant for Local Developer Testnet milestone so it will be dealt with later.

3 Likes

I don’t know enough about cryptography to be able to authoritatively asses this but we are already truncating the public inputs hash inside the Decoder so if it’s ok in this case as well truncating seems like a more consistent and cheaper option.

3 Likes

There is a difference in where this “truncate” happens. We are truncating only the last hash in the decoder link you sent throughout the tree generation etc. We are using just normal sha256.

4 Likes

I have a faint memory that truncating the output of a sha256 hash makes cryptographers very unhappy. We should check with the crypto team the best thing to do.

3 Likes

In the above case, what you need is the “construct” to provide integrity of the logs. Therefore, we need collision-resistance (and therefore secure against second pre-image attack). If you can find two inputs of the above construct with the same output, this would mean that you found a very bad property of SHA-256 as one would be able to find SHA-256 values of 2 different inputs which have essentially most of the bytes correlated. Equally important as a cryptography engineering practice is that the data structure (that is hashed) is serialized through an injective function (big issue while appending variable-size arrays for instance. More on this, Horton Principle and Pascal Junod’s post). Namely, you need to make sure that the collision resistance applies at the original data structure (and not the serialization).In the context of a Merkle tree, one also need to serialize the leaves differently than the inner nodes as explained here Note that making a modulo operation and then hashing would be totally broken. Finally, this construct has the undesirable property of having a bias, namely if P is not an integer divisor of 2^256, the output will definitely not uniformly distributed over 0…P-1. The question is whether it is an issue in this context?

8 Likes

Actually, there is a NIST document (See Section 5.1) which covers truncation of a hash function output and how to do this in a standard way (discard right-most bits). This is basically fine as long as the length of the truncated output is adequate for the security level you need (for instance, truncating to 220 bits would still offer 110 bits security against collision resistance attack.)

Note that SHA-224 resp. SHA-384 is SHA-256 resp. SHA-512 truncated to 224 resp. 384 bits and there is a SHA-512/256 hash function consisting on truncating SHA-512 to 256 bits.

Bottom line, we could really truncate SHA-256 to 253 bits and fit such hash value into BN254.

Of course, using a more ‘modern’ hash function such as SHA-3 or BlakeXXX or more ZKSnarks friendly is worth considering. (Seems we are constrained by solidity hash function support here.)

8 Likes

I think we need to support the emission of all of the things you list. I think submitting them all via the proposed log_hash field is nice and tidy. But it would need custom log processing functions, to work nicely. :heart:

5 Likes