In our open source manuals there is plenty of text about the operation of our silicon. But that can be a little dry. One of the areas that people struggle with the most even after reading the manuals is how the data flows into and out of our chips. I’ll be talking mostly about the 1 Gigabit stuff in this article, but the basic flow is the same for 10G, only the locations and names might be different. As always with these chalk talks, if the manual and the blog disagree, the manual is always right, the spec update doubly so. Now we start the dance by talking about dogs.
Our silicon has always had a head and a tail. And when you talk about tails, almost like it was a law, people use the dog chasing its tail analogy. One debug warrior was even called “Dogcatcher” as a title. If you have seen a dog chase its tail, it is strangely cute as a button. Probably not so much fun for the dog, and just to show I’m not a dog hater, my wife’s cat chases its own tail too. But this isn’t a blog about dogs and cats. So here is my attempt at explaining it without using the dog analogy. We’ll see how I do.
Ethernet is all about moving data from applications on a host machine to anywhere in the world, or above it. The data resides in main system memory, and we believe it should stay there. We use descriptors to point the hardware at the data in main memory. We also include handling instructions like stateless offloads to use, VLANs tags to insert and size of the data. This also includes a location of the data. In our 10/100 line it was a linked list and this would cause troubles because of a race condition where hardware could read faster than software could write and both sides could have stale data, and worse make bad decisions off of that stale data. In our 1 Gigabit line, and our 10 Gigabit side too, we use an array of descriptors to be an invoice to the hardware. The descriptors are also in main memory, which speeds operation. The Base Address registers (RDBAL/H and TDBAL/H) point to the start of the array and TDLEN/RDLEN state the size of the invoice. The transmit side (starts with a T in the registers) and the receive side (starts with an R) are very close in their operation. Once you understand one, you get both.
So how does the HW know which member of the array to go fetch or retrieve. Here is where Tail and Head come into play. Each is an index into the array. Tail is “owned” by software and Head by hardware. Software moves the Tail, hardware the Head. They are used in a circular fashion, so if you have a Tail value of 4 and you write a Tail value of 8, hardware will use 4, 5, 6 and 7. From Head to Tail is owned by the hardware and is ready to be processed, and from Tail to Head is owned by software and is already processed. On the Transmit side, when Tail is equal to Head there is nothing to do. On the receive side, tail should never be equal to head. This means software has to be careful not to accidently write Tail past the Head. The Head on either side of the Ethernet Controller should never be written during normal operation, and if set at init time it should only be set to zero. Starting at a non-zero init value will produce non-deterministic results. So don’t try it.
This makes a bus trace very interesting to read. First you will see the Tail write happen to the device. It will then fetch the descriptors out of the array as pointed to by the BAL/H registers. It will then process the descriptors one at a time, even thought it might fetch several descriptors. This will generate another series of transactions related to the buffer. On the RX side it will write the data out, on the TX side a read to be sent on its way. Then if configured there will be a writeback of the descriptor, updating it with information on what happened. Done bits, size of the data received that type of stuff. Then Head moves towards the Tail, and the next array element is processed. This is where the discussions really start, because if it’s a true “Tail” why does “Head” move towards it. And I can’t comment on that further without violating the “no dogs” rule I set earlier. But once you get used to our usage, it enables you to write software for all of our 1 Gigabit and 10 Gigabit products.
Time to wrap up before I can’t control my dog references.
1) 1) Head follows the Tail. Tail is moved by software.
2) 2) Descriptors are the key to data movement into and out of our controllers
3) 3) Thanks for using Wired Intel® Ethernet