Reading Sangod's recent post on write acceleration, I couldn't help writing a response, as I've been looking at this whole subject recently.
First of all, I don't disagree with the concept of synchronous. Yes, the I/O must be confirmed (key word there) at the remote and local site before the host is given acknowledgement of the I/O being complete. Typically, enterprise arrays will cache a user I/O, issue a write to the remote array (which will also be cached), acknowledge the I/O to the host and destage at some later stage.
Sangod mentioned two techniques; cranking up buffer credits and acknowledgement spoofing. Buffer credits are the standard way in which the fibre channel protocol manages flow control. As each FC device passes data (for example an HBA to a switch) then the device can keep sending packets of data whilst the receiving device issues the R_RDY signal back. Buffer credits are essential as FC packets take a finite amount of time to travel a fibre optic cable. The longer the distance between devices, then the more packets which must be "on the line" in order to fully utilise the link. The rule of thumb is 1 buffer credit for each km of distance on a 2Gb/s connection. So if you don't have enough buffer credits, then you don't make best use of the bandwidth you have available. Having lots of data in transit does not compromise integrity as nothing has been confirmed to the originating device.
Moving on, there are devices which can perform write acceleration by reducing the SCSI overhead by acknowledgement spoofing. I've taken the liberty of loaning a couple of graphics from Cisco to explain how a SCSI transaction works.
When a SCSI initiator (source device) starts a write command, it issues a FCP_CMND_WRT to the target device. This is confirmed by the target with an FCP_XFER_RDY. The initiator then issues data transfer (FCP_DATA) repeatedly until all the data is transferred. The target confirms successful receipt of all of the data with FCP_RSP "Status Good". The initial preamble as the write request is started can be reduced by the switch connected to the initiator issuing an immediate FCP_XFER_RDY, allowing the initiator to immediately start sending data. The data transfer and the FCP_CMND_WRT operate in parallel, saving the time of this part of the exchange. No integrity is risked as nothing is confirmed by the target until all data is received.
First of all, I don't disagree with the concept of synchronous. Yes, the I/O must be confirmed (key word there) at the remote and local site before the host is given acknowledgement of the I/O being complete. Typically, enterprise arrays will cache a user I/O, issue a write to the remote array (which will also be cached), acknowledge the I/O to the host and destage at some later stage.
Sangod mentioned two techniques; cranking up buffer credits and acknowledgement spoofing. Buffer credits are the standard way in which the fibre channel protocol manages flow control. As each FC device passes data (for example an HBA to a switch) then the device can keep sending packets of data whilst the receiving device issues the R_RDY signal back. Buffer credits are essential as FC packets take a finite amount of time to travel a fibre optic cable. The longer the distance between devices, then the more packets which must be "on the line" in order to fully utilise the link. The rule of thumb is 1 buffer credit for each km of distance on a 2Gb/s connection. So if you don't have enough buffer credits, then you don't make best use of the bandwidth you have available. Having lots of data in transit does not compromise integrity as nothing has been confirmed to the originating device.
Moving on, there are devices which can perform write acceleration by reducing the SCSI overhead by acknowledgement spoofing. I've taken the liberty of loaning a couple of graphics from Cisco to explain how a SCSI transaction works.
When a SCSI initiator (source device) starts a write command, it issues a FCP_CMND_WRT to the target device. This is confirmed by the target with an FCP_XFER_RDY. The initiator then issues data transfer (FCP_DATA) repeatedly until all the data is transferred. The target confirms successful receipt of all of the data with FCP_RSP "Status Good". The initial preamble as the write request is started can be reduced by the switch connected to the initiator issuing an immediate FCP_XFER_RDY, allowing the initiator to immediately start sending data. The data transfer and the FCP_CMND_WRT operate in parallel, saving the time of this part of the exchange. No integrity is risked as nothing is confirmed by the target until all data is received.
I see no issue with this kind of spoofing as the initiator is not being told about the completion of the I/O before it has actually happened. What I do see as a concern is that the target may not be able to accept the data and therefore if the write fails, the source needs to be able to roll back if that happens.
In terms of getting best performance, multiple replication groups (e.g. multiple RA groups) would make best use of any WA technology. Cisco publish stats which show that. So WA could be good and safe.
Posts
