Ring.onReadEnd and Ring.ReadDiscard call Reset() when the last readable
byte is consumed, which rewinds Off and End to 0. That is a valid empty
state, but it also moves the write position, and bytes staged past that
position with PeekWrite are addressed relative to it. After the rewind a
Commit hands back whatever bytes happen to live at the start of the
buffer instead of the staged ones.
tcp stages out-of-order segments exactly this way (see reassembly.store,
"the ring write pointer advances in lockstep with rcv.NXT, so the staged
bytes are always where seq implies"). Reading is driven by the
application, so a receiver that drains its stream while a gap is open
loses the staged tail silently: the stream arrives with the correct
length and the wrong contents.
Observed on a Sophgo SG2002 (100Mbit DWMAC, receive ring dropping frames
under load): a 16 MiB download arrived complete with a different
SHA-256, and TLS over the same path failed with "bad record MAC", while
the transmit path was bit-perfect.
Mark the ring empty without rewinding instead: End=0 is what empty
means, and the write position is then Off, which Ring.Write already
supports explicitly.
Two tests, both failing before the change:
- internal: staged bytes survive the ring being read empty.
- tcp: a reassembled stream is byte-identical when segments arrive
reordered (segment K arrived as the contents of an earlier segment).
Co-authored-by: Derek den Haas <i.pestano@easyflor.nl>
* add http/httphi
* begin adding httphi tests
* claude found neat bugs
* add low level Handle function and more tests
* more tests, run go generate
* add Hijacker-like functionality
* improve locking and acquisition of Exchanges in reconfiguring
* several bugfixes, add internal.IntLen, round up http-linux example with new router API
* small nit
* add benchmarks
* add query handling
* remove ForEach pattern, allocates in TinyGo
* massive documentation push and code reordering in files
* Router.Handle returns error after being torn down
* run go fix
* rework Mux interface to receive a string request path
* add MethodFrom
* minor doc nit
* fail on incomplete staging
* add raw buffer access
* add streaming API distinct from Exchange
* begin adding multipart form logic
* finish rounding up multipart form parsing
* remove status type
* first Multipart approach
* begin adding readMultiPart
* add Exchange.ReadMultiparts reimagining of clanker slop
* ai insists with backoffs
* simplify clanker slop
* apply go fix
* add a pattern argument to Mux
* explicit header key/value alloc and add ExchangeConfig
* fix tests after excplicit header alloc change
* fix examples
* run go fix
* expose rawsock as experimental package (will use for external benchmarks)
* remove backoff from form parsing
* @MDr164 suggestions get potential fixes
* apply go fix
* add examples
* add README.md
* fix rawsock tinygo implementation
* apply @MDr164 various fixes
* update documentation on ContentLength methods and fix bug in Form reset on empty body
* fix tests
* add fuzz tests
* run go fix
* io.ErrNoProgress on parsing form spin
* run go fix
* remove backoff assumption from Router
* httphi.Handle rejects unsupported protocols
* go format router.go
* add kvbuffer
* rewrite Cookie with KVBuffer
* mid refactor of KVBuffer into Header
* work on KVBuffer exhausted semantics
* add Go's ServeMux Request.PathValue access semantics to Exchange, Mux and MuxSlice
* add PathValue example
* document all the things; improve req Query semantics; add Form.EnableBufferGrowth
* unexport kvBuffer
* add Exchange.PathValueAppend
* use stdlib in example instead of rawsock
* remove rawsock from http example
* add darwin arch rawsock
* fix example
* rename Router.TeardownGoroutines to Shutdown matching http.Server.Shutdown
* rename types and identifiers
* @MDr164 Content-Type and Transfer-Encoding bug catches
* feat(tcp): add out-of-order segment reassembly
Add an opt-in, bounded out-of-order reassembly buffer so a single lost
segment can be recovered by retransmitting the gap while later segments
are held and delivered once the gap fills.
The receiver also subtracts buffered out-of-order bytes from the
advertised receive window and avoids challenge-ACK aborts for in-window
future data. Reassembly is disabled by default.
Generated with LLM assistance.
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
* implement review feedback around rx buffer reuse
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
---------
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
This adds a small in-repo tool and updates the ci.yml file in order
to make benchmark and most importantly allocation results visible in a
PR. It also drops the 3rdparty action we had commented out.
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
* use ltesto as package to contain scheduler/goroutine testing logic
* add sched usage to another test
* testCloseTransmitsPending rewrite
* replace deadline with context
- http/httpraw: cap header slice growth to pre-allocated capacity, fix
benchmark to reuse Header across iterations (0 allocs/op)
- internal/ring: replace TODO panic comments with invariant explanations
- tcp/conn: document truncated-frame offset check
- tcp/handler: replace TODO with net.ErrClosed on RST-closed connection
- internet/stack-udpport: filter incoming packets by remote IP address
when configured via SetStackNode; skip filter for IPv4 multicast
destinations (class D) so mDNS and similar protocols work correctly
Generated with LLM assistance.
Signed-off-by: Marvin Drees <marvin.drees@9elements.com>
* start working on tracking down tcp buffer bug
* mtu refactor
* modularize test
* more precise testing
* tests fail, but is it the failure we are looking for?
* fix typo in espradio link (#76)
* implement a new backoff abstraction (#75)
* rewrite backoff api
* rewrite tcp.Conn.Write
* keep fixing small things
* much better Conn.Read implementation
* fix critical overflow bug in internal.ConnRWBackoff
---------
Co-authored-by: Joel Wetzell <jwetzell@yahoo.com>
* begin adding better fuzz test
* finish adding working fuzzer
* implement mutateipv4
* fuzzer finds a panic in ICMP client on receive empty payload
* rename fuzz tests to reflect fuzz methodology
* begin adding icmp client
* rely on anon structs
* add tests
* tests passing
* icmp fleshed out
* rework icmp to include ip addr
* rename StackAsync.Demux/Encapsulate to RecvEthernet and SendEthernet
* remove legacy unreachable TCP tests
* remove uses of deprecated internet.StackNode in preference of lneto.StackNode
* documentation
* rename methods to signal no I/O happening
* tcp: remove retransmit logic entirely; add duplicate ack counting to ControlBlock
* retransmit implemented in nice simple straightforward way
* narrow down retransmission cases
* remove old timing tests
* add ControlBlock retransmit test
* add failing handler test
* reworking payload length semantic meaning in code
* fix establish conn logic
* clean up tests and add TCB dupack generation and test it
* catch pending retransmit satisfy in test
* add fuzz test for control block
* bugfix: be more strict in what is considered dupack
* add IncomingIsDupACK docs
* limit queue of retransmits
* protect retransmit overflow from incorrectly updating nxt
* add mdns
* define mdns.Client
* fix some mdns stuff
* refine dns package for use with mdns
* remove Querier and Responder and replace with Client
* add record setting methods on dns.record to reuse record buffer
* protect against unbounded client answer growth
* round off sharp mdns edges; reduce allocs
* add mutlicast to stacks; add xnet tests for mdns
* fix documentation on acceptmulticast field
* mdns working
* pcap: reuse Frame memory
* slog: reduce heap allocations of addresses; also prevent heap alloc of dhcp options in pcap
* dns: heapless improvement; add StackAsync buffer for more heapless operation; start thinking of errors
* errors: begin standardise errors in lneto
* errors: finish standardization of errors
* fix merge issues
* add more lneto errors to rest of package
* format errors.go
* reduce heap allocations in tcp logging; omit use of AppendFloat which allocates a metric sh*tton
* debugheaplog: better heap statistic logging
* heap: use string for pcap.Frame.Protocol
* add potential to eliminate Flags.String heap alloc, remove incorrect HEAP comments
* add StackAsync.DebugErr and httpraw.SetBytes
* many heap alloc reductions and replacement of bytes.Equal with internal.BytesEqual
* pcap: reuse Frame memory
* slog: reduce heap allocations of addresses; also prevent heap alloc of dhcp options in pcap
* dns: heapless improvement; add StackAsync buffer for more heapless operation; start thinking of errors
* reuse function for reclaiming frames among all protocols for lower memory usage
* remove extraneous code
* claude suggests a way forward
* add timing to capture printer
* add pcap.Flags
* fix ICMP CRC calculation and add test
* bugfix: still send data on half-close state(close-wait)
* fix pcap test
* Simplified checksum calculation and verification
* Fixed tests
* UDP: using NewBoundedFrame to create a Frame instance limited to the actual frame size.
* Minor: renamed some functions
* Commented and made more readable consecutive SetCRC calls.
* Fixed icmp checksum calculation after rebase
* Replaced NewBoundedFrame with explicit validation
* Repeating select if it is interrupted
* Restrict retries for syscall.EINTR to 10
Limit the number of retries on syscall.EINTR to 10.
---------
Co-authored-by: Pat Whittingslow <graded.sp@gmail.com>
* do not rely on unspecified Go behaviour; fix TCP ring buffer bug
* even more stricter error handling on tcp connections
* stricter lock copying in tcp.Conn, even though likely not a problem
* add listener and tcp pool logging
* forgot reqAddr unused
* add logging to TCPConn and friends
* begin reviewing TCP listener
* begin adding listener tests
* adding TestExchange implementation
* split out legacy and Step tests
* move listener to tcp package
* fix up tcp tests taking very long
* add xnet.TCPPool and work on TCPPool semantics
* xnet: TCPPool only accepts established connections
* xnet: getting to bottom of panic in xnet.Listener
* xnet: improve listener tests
* fix several data races in testing fixtures
* fix more synchronization things in listener test
* finalize tcplistener test
* add prints to investigate slowness
* add loop sleep for blocking Stack
* add connection close http attribute and detect end of HTML page
* add timeout to bridge read
* examples/xnet: add ntp timestamp to prints and show output in readme
* add polling to http tap interface
* more digits in ntp format
* prevent crashes on non-8 timestamp
Derek den Haas
Pat Whittingslow
Marvin Drees
ddirect