(2.le;PN-161 - Using the DR11-W DMA device for Interprocessor Communications in HRT-11. R.le;PN-162 - An RSX-11M Device Driver Implementing Network Protocols on the DR-11W. \.els f.sk 1 pThe purpose of this document is not to discuss the whole framework of zinterprocessor and inter-task communication but rather to list the functions performed by the CD driver.The document is in a similar form to the RSX-11M I/O driver's reference manual.  All information necessary to interface directly with the CD driver can be found in the sections following. .sk 1 .hl 1 Get LUN Information Macro ^& Word 2\& of the buffer filled by the Get LUN Information system directive contains the first device characteristics word. This word is 0 for the CD: device. .b;^& Word 3\& contains the driver State word (shown in octal) .sk 1 .literal  1 = IDLE No transfer of message in progress  2 = ASTQUE Data transfer in progress - awaiting  receiver's buffer  4 = RECPEN DMA in progress receiving data block $ 10 = WTLACK Link bid initiated, waiting link acknowledge . 40 = WTPTC Have acknowledged link bid, waiting for PTC 8 200 = WTSGAK Have sent Signal, waiting for Signal B acknowledge L 400 = WTRCWC Have sent word count, waiting Receivers word V count ` 1000 = TRANSF DMA in progress sending data block j 2000 = WTXMWC Have sent PTC acknowledge, waiting for word t count ~ 4000 = DOWN Link down  10000 = STWAIT Waiting for Start acknowledge  60000 = FORKED Waiting to execute, or executing Fork level  code .end literal .sk 1 .b;^& Word 4\& contains the total number of link level protocol starts or re-starts which have been attempted by the driver. When the other end of the link is not responding to link start requests (either turned off, or not running DR11-W link software) this number rapidly becomes large. Once the link is "up" this number is incremented by one only when a failure in the link protocol occurs. .b;^& Word 5\& is not used by the driver at present. In future it may be used data in one direction. It corresponds to the fourth device characteristic word, whose value can be modified by the MCR command SET /BUF=CDn: .sk 2 (.hl 1 Logical Units and CD unit numbers 2Each physical link (DR11-W to DR11-W connection) has a different CD device <unit number. FIf the driver is assembled with the intraprocessor option then Pthe internal link is the highest CD unit number. Z A task may assign logical units to CD device units in the normal way. In dprincipal more than one LUN can be assigned to the same CD unit number. nThe driver's data bases however, are on the basis of CD unit numbers, not xtask logical units. When a task declares which types of messages it intends to receive this information is correlated with the CD unit number of the LUN on which the declaration is made. Similarly Read requests on a LUN are correlated with the CD unit number and type of message rather than the LUN number. This means that if multiple LUNs are assigned to the same CD unit number, cancellation of I/O requests such as IO.KIL and IO.CLN made on any one of the LUNs will cancel all outstanding request for a particular CD unit number, regardless of which LUN those requests were made on. .hl 1 QIO Macro Table 1 lists both the standard and non-standard functions of the QIO macro which are valid for the CD driver. .sk 2 ^& .center;Table 1 .center;QIO functions for CD Driver \& .literal " ,------------------------------------------------------------------------ 6 @ Format Function J T------------------------------------------------------------------------ ^ hQIO$C IO.SPA,...., Declare Packet type code affinity r |QIO$C IO.RPA,...., Remove Packet type code affinity  QIO$C IO.RLB,...., Read logical block  QIO$C IO.WLB,...., Write logical block  QIO$C IO.SIG,...., Send signal  QIO$C IO.REJ,.... Reject data block awaiting  receiver buffer space  QIO$C IO.STU,...., Return Driver Status  QIO$C IO.KIL,.... Cancel IO.RLB,IO.WLB and IO.SIG  queued I/O requests  QIO$C IO.CLN,.... Remove all packet type code & affinities 0 :QIO$C IO.AWA,.... Restore driver ownership of D DR11W interrupt vector N------------------------------------------------------------------------ X bwhere: l PTC is the packet type code value. This must be a value v between 0 and 255.  astadd is the address of an AST routine which will be entered  whenever a signal for the specified PTC arrives, or  when a sender wishes to send a data block with the  specified PTC, but no IO.RLB function has been issued.  The AST address must be within the task's address space.  stadd is the starting address of a data buffer. The address  must be word aligned.  size is the size of the stadd data buffer in bytes. The  specified size must be even, non-zero and will be  interpreted as an unsigned integer between 1 and 65,534  The buffer muxt be within the task's address space.  signal is a data word, the low byte of which will be  interpreted as a signal value.   .end literal .hl 2 Subfunction bits *Three subfunction bits are used by the communications driver. 4Table 2 shows which QIO functions each of the subfunction bits may be applied >to. Any valid combination of subfunction bits may be used on an I/O function Hcode R.sk 2 \^& f.center; Table 2 p.center; QIO sub-functions for CD driver z\& .sk 1 .literal  -------------------------------------------------------------------------  Subfunction May be used with Description -------------------------------------------------------------------------  SF.SLP IO.RLB,IO.WLB On successful (or truncated) completion  of the I/O function, driver  must 'sleep'. This means surrender  control of the DR11W device interrupt  vector.  SF.RSW IO.RLB,IO.WLB,IO.SIG Resource wait subfunction. The I/O  request will be queued even if the  link is not 'UP'. It will await $ successful startup of link protocol. . SF.RDQ IO.RLB The I/O request will be queued unless 8 it is needed for a data block already B in the process of being received. L IO.RLB packets which do not have this V subfunction bit will terminate with ` an error status unless they do j pertain to a pending data block, t awaiting a receiver buffer. ~ ------------------------------------------------------------------------- .end literal .hl 2 IO.SPA - Set packet type code affinity This I/O function is used to declare which type of messages a task wishes to receive or to be notified of the arrival of. When a task successfully issues an IO.SPA function for a packet type code it establishes an "affinity" for messages with this PTC, for a particular CD unit number. This affinity or ownership of a particular PTC then ensures that all messages arriving over the link with this PTC destination identifier, are passed to the owning task. A task may not issue an IO.RLB QIO function for a particular PTC, without first having established an affinity for that PTC. The IO.SPA function call must specify the address of an AST routine as the second parameter. This AST routine will be entered whenever a signal for  the specified PTC arrives, or whenever a sender on the other machine wishes  to write a data block with this PTC, to the link, but no IO.RLB function has been issued ( and queued for this PTC. 2 A task may issue the IO.SPA function for an arbitrary number of packet type < codes. Any number of IO.SPA function calls may have the same AST address F specified. The task can distinguish between different PTC's from the P parameters passed to the AST on the stack. Each successful issuance of an Z IO.SPA function leads to the creation of a seven word in-pool data structure. d Once a task has established a packet type code affinity for a particular n PTC, on a particular CD device unit number, IO.SPA functions for that same x PTC issued by other tasks will terminate with an error status. The IO.SPA QIO function terminates immediately leaving the task awaiting incoming messages, but without pending I/O, so therefore potentially checkpointable. This function is handled even if the physical link is not "up". .hl 3 Pending message AST routine On entry to the AST routine the stack contains several parameters, which must be removed before exiting the AST. The stack contents are: .b; .literal SP+6 Physical unit number of DR11-W with pending incoming message. SP+4 If positive is the Word Count of the pending incoming data block, sender is asking to send.  If negative the low byte of this word is a signal value  SP+2 Packet type code of pending incoming message  SP+0 Stack displacement to be added to SP on exit to remove " all parameters from stack , 6 .end literal @ The incoming message may be either a signal or a data block. The receiver J does not need to issue a QIO to receive the signal since it has already been T passed as a parameter on the stack. If the AST ^ has been entered to notify the task that a sender is trying to write a data h block there can be several possible outcomes. The data block to be sent is r already 'pending' when the AST routine is entered, which means that the link | is busy and no further messages can be transferred over the link, in either direction, until this particular transfer terminates. The ways in which it may terminate are: .ls .le;Task may issue IO.RLB QIO function to receive the full data block. .le;Task may issue IO.RLB QIO function with smaller word count than that passed as a parameter on the stack. Truncated data block transfer will take place. .le;Task may issue IO.REJ QIO function to indicate that it does not wish to receive this message, at this time. .le;Task may take no action. After approximately 2 seconds the pending data block transfer will terminate with a time-out error status. NOTE: during this 2 seconds the link will not be able to be used for other transfers, so this is not a recommended course of action.  .le;Task may issue IO.RLB QIO function but an error condition may result  (see table of errors in Table 5).  .els & .hl 2 IO.RPA - Remove packet type code affinity 0 The IO.RPA function removes a previously declared ownership (affinity) of : a particular packet type code. If IO.RLB QIO functions have been issued D and queued for this particular PTC then the IO.RPA function will terminate N with an error status. X .hl 2 IO.RLB - Read logical block b The IO.RLB function is used to read a logical block of data from a link l (either a DR11-W interprocessor v link, or the internal intraprocessor link). If the SF.RDQ subfunction bit is not set then this function will succesfully read a block of data only if it is issued to read a pending data block, by the task owning the packet type code of that data block. If the SF.RDQ subfunction bit is set then this function will successfully read a block of data whenever a data block for that packet type code is next sent over the link. (This may be immediately if there is actually a pending data block when the QIO is issued). The "size" parameter on the IO.RLB QIO specifies the maximum block size that may be read. If this is smaller than the size specified by the sender of a data block (e.g. on the IO.WLB QIO ) then only "size" bytes of  data will be transferred from the sender's buffer to the receiver's buffer. The IO.RLB QIO function will terminate with  a Data overrun error status. See table of error statuses in Table 5 for this and other possible error status returns. * .sk 2 4 .hl 2 IO.WLB - Write logical block > The IO.WLB function is used to write a block of data to the link (either H a DR11-W interprocessor link, or a intraprocessor link) R Any legal packet type code may be specified to identify \ the destination of the block of data. There are several possible f outcomes when a task issues an IO.WLB QIO function: p .ls z .le;The packet type code specified is not owned by any task/program on the destination machine. An error status is returned from the QIO. .le;The full data block is read into a receiver's buffer. .le;Part of the data block is read into a receiver's buffer. A data overrun error status will be returned by the IO.WLB QIO. .le;The packet type code specified is owned by some task/program but the data block is not read from the link into a receiver's buffer (e.g. with a IO.RLB QIO function for an RSX-11M destination system) within the timeout period. The timeout period is approximately 2 seconds. .le;The packet type code specified is owned by some task/program but the data block is rejected by that owner (e.g. with an IO.REJ QIO function for an RSX-11M destination system). .le;Other error condition (see table of errors in Table 5). .els .sk 2 $.hl 2 IO.SIG - Send Signal .The IO.SIG function is used to send a one byte value (a Signal) to the 8owner of the specified packet type code. If the specified packet type Bcode is not owned the IO.SIG function will return an error status. LIf the specified packet type code is owned the Signal will be Vsuccessfully sent over the link to the receiving program. `.sk 2 j.hl 2 IO.REJ - Reject Data Block tThe IO.REJ function may be used by a task when a sender wishes ~to write a data block with a packet type code owned by the task. A task when declaring a packet type code affinity implicitly states that it wishes to receive all messages with that packet type code. In normal operation, a task which is informed of a pending data block by the AST mechanism described above, will proceed to read the data block into a buffer. It may however instead choose to reject receipt of the data block, perhaps because of insufficient buffer space. The reject mechanism allows the pending data block transfer to be terminated immediately and the link thus freed for further messages. .sk 2 .hl 2 IO.STU - Get Driver Status This function may be used to obtain the driver's current state. The state word returned is the same as Word 3 returned by the Get LUN information .sk 2 .hl 2 IO.KIL - Cancel all Reads and Writes to Link The IO.KIL function will cause cancellation of all queued IO.RLB, IO.WLB (and IO.SIG QIO functions, for the CD unit number corresponding to the LUN 2on which the IO.KIL is issued. The IO.KIL function does not affect an <IO.RLB, IO.WLB or IO.SIG function which is actually currently being processed Fby the driver. This will terminate normally or timeout within the normal Ptimeout period. Z.sk 2 d.hl 2 IO.CLN - Cancel all Packet Type Code affinities nThe IO.CLN function will cancel all packet type code affinities for the CD xunit number corresponding to the LUN on which the IO.CLN is issued. However the IO.CLN function will only successfully carry out this cancellation if there are no queued IO.RLB QIO functions. Normally the driver is called with this particular IO function only by the executive, during Task run-down, and after an IO.KIL has already been processed. .sk 2 .hl 1 Link Hibernation Services .hl 2 SF.SLP - Sleep subfunction The CD driver will disconnect from the DR11W device interrupts, thus allowing a privileged task to connect to interrupt, on succesfull processing of a read or write logical block function which has the SF.SLP subfunction. In order to properly interlock the passing of control of the link to two privileged tasks, one in each machine, a proper protocol between the two tasks must exist, coordinating the use of "write block of data and sleep" QIO function with the use of "read block of data and sleep" function on the receiving end of the link. By this means control of the link hardware can be passed from the two device drivers to the two special link tasks with all other queued "I/O functions remaining queued by both drivers to await return of the link ,control to the driver. The passing of control at both ends of the link is 6in fact accomplished within a single data block transfer. @.hl 2 IO.AWA - Awaken CD driver JWhen a privileged task has relinquished its connection to the DR11W interrupt Tvector the CD driver must be informed that it may regain control of the ^device, restart the link protocol and continue processing its queued I/O hrequests. The IO.AWA function is used for this purpose. If the device rinterrupt is still connected to a task when this function is issued then |an error status return will be given. .sk 3 .hl 1 Programming Hints .hl 2 Vertical Parity Support and Data Validation The DR11W does not support vertical parity. In all our tests on transfers of blocks of data of varying sizes across a DR11-W to DR11-W 50 ft. link we have had no instances of errors in the data. .hl 2 Retry of unsucessful transfer The transfer of a signal or block of data from the sender to a receiver may fail for a number of reasons. For example there may be no suitable receiver, the receiver may fail to issue a read for the data within a timeout period or the DR11-W device itself may assert an error condition such as non-existant memory. The CD driver never tries to re-transmit a message whose transfer has failed, whatever the reason for the failure. The failure is reported as an error status to the sender of the block, and also to the receiver if the transfer had reached a stage where an actual &IO.RLB QIO function was being processed. 0Certain types of error status returned to the sender of a message are such :that the sender knows that the receiver has no information about this Dmessage, in fact there is no receiver. NOther types of error status returned to the sender indicate that the Xreceiver exists but has either failed to read the data or chosen to reject bthe data. lThe final type of error status involves both the sender and the receiver. vData transfer from the sender buffer to the receiver buffer is actually attempted, but the transfer fails in some way. This latter type of error is extremely rare. It is not an expected statistical feature of the hardware. Recovery from all such errors is handled by the CD driver. .hl 2 Speed and Space considerations The driver may optionally be assembled with a debug trace feature. This takes up approximately 800 words for a driver with an actual DR11-W link. The trace feature even if present in the driver need not be enabled. There are about 3 milliseconds of additional overhead on each transfer when the trace option is enabled.  Each declaration of a packet type code affinity causes a 7 word pool block to be allocated by the driver. These pool blocks are linked together and when a message arrives over the link the linked list of pool blocks is searched for an appropriate packet type code. Obviously, use of many different packet type codes would use up a large amount of pool space and also affect the speed of data transfers marginally. .hl 2 Full Duplex - Half Duplex considerations *Although the physical DR11-W link can only be used in one direction at a time, 4the CD driver allows the link to be considered as a full duplex device. A >task may have any number of IO.RLB, IO.SIG and IO.WLB outstanding at one time. HIf the driver has a queue of IO.WLB Ror IO.SIG requests at the time it is terminating \the read of a block of data from the link (IO.RLB function) then the link flevel protocol allows the driver to take ownership of the link to start a pmessage transfer. This ensures fair bi-directional use of the link and is znecessary because of differing processor speeds. .hl 2 Redirection of Device CD units are not redirectable .hl 2 Unloading a Loadable driver ^&DO NOT\& unload a driver which supports physical DR11-W devices. System one-shot timers may be active and a system crash could result. The next version of this driver will attempt to protect against this. .hl 1 Powerfail The CD driver will attempt to re-start the link protocol on powerfail. However correct handling of an I/O packet which is being processed at time of powerfail is not assured. .hl 1 Sysgen Options required The CD driver needs rotating data light support in the RSX-11M system. The idle counter is used as a random number source in the link startup protocol.  For maximum speed task to task message transfer over the internal link the largest vector for $BLXIO should be specified  The driver does not currently run on a 22-bit mapped PDP-11. $.hl 1 IO Function and Sub-function Codes .Tables 3 and 4 give the values (in octal) for all the I/O functions and sub- 8functions used by the CD Driver. For convenience both standard I/O functions Bsuch as IO.RLB and IO.KIL and driver specific functions are listed in one Ltable. `^& j.center;Table 3 t.center;CD Driver IO Function Code values ~\& .sk 1 .literal  --------------------------------------------------------------------------   Code Word Equivalent Code (high byte) Meaning  --------------------------------------------------------------------------  IO.SPA 014400 31 (25.) Set packet type code  affinity   affinity  IO.RLB 001000 2 (2. ) Read logical block ( 2IO.WLB 000400 1 (1. ) Write logical block < FIO.SIG 017000 36 (30.) Send Signal P ZIO.REJ 017400 37 (31.) Reject pending data block d transfer n xIO.STU 014000 30 (24.) Get driver state  IO.KIL 000012 0 subcode 12 Cancel all IO.RLB,IO.WLB  and IO.SIG queued I/O  IO.CLN 000007 0 subcode 7 Remove all packet type code  affinities  IO.AWA 016400 35 (29.) Restore driver ownership of  DR11-W after sleep  subfunction  -----------------------------------------------------------------------  .end literal .page ^& ".center;Table 4 ,.center;CD Driver I/O sub-function values 6\& @.sk 1 J.literal T----------------------------------------------------------------------- ^ h code value meaning r |-----------------------------------------------------------------------   SF.SLP 1 Sleep i.e. disconnect from DR11-W interrupt  vector   SF.RSW 2 Resource wait   SF.RDQ 4 Queue Read function  ----------------------------------------------------------------------- .end literal .sk 4 .hl 1 Status Returns The error and status conditions listed in Table 5 are returned by the communications driver. Values are given in decimal. .sk 2 .tp 9 &^& 0.center;Table 5 :.center;CD Driver Status Returns D\& N.sk 1 X.literal b----------------------------------------------------------------------- l v Code Reason  -----------------------------------------------------------------------  IS.SUC 1 Successful completion   The operation specified in the QIO directive was completed  successfully. If the operation involved reading or  writing of a data block, the second word of the I/O  status block can be examined to determine the number  of bytes transferred.  IS.MAS 2 Link relinquished by driver with link master status   After a Read or Write I/O function with the sub-function  code SF.SLP has been successfully terminated this status  is returned.  The driver returning this status had the status of * 'master' as defined in the link protocol. This means that 4 the program who is gaining control of the link need not > necessarily re-start the link protocol at this time. H Truncated data transfers are also treated as successes R and this status code is returned for them on a sleep \ sub-function. The second word of the I/O status block f can be examined to determine the exact number of bytes p transferred. z IS.SLV 4 Link relinquished by driver with link slave status   As for IS.MAS status except that the driver had the  status of 'slave' as defined in the link protocol.   IS.PND 0 I/O request pending   The operation specified in the QIO directive has not  yet been executed or completed. The I/O status block is  filled with zeros.  IE.IPT -112. Illegal packet type code value   Valid packet type code values are between 0 and 255.  $IE.PTO -100. Packet type code owned . 8 The packet type code for which a set affinity or B read function has been specified in the QIO directive L is already owned, and in the case of a read function, V by another task. For a set affinity function the packet ` type code may either be owned by another task or j already owned by the issuing task. t ~IE.DNE -103. Destination does not exist   The packet type code on a write block or send signal  type QIO function does not have an owner on the  destination machine.  IE.NRP -104. No receive of data block pending   When a IO.RLB function is specified in the QIO directive,  without the SF.RDQ subfunction code, this error status  will be returned unless the IO.RLB function is the from  the correct task and for the correct packet type code  for a block of data which is 'pending' awaiting a read  IE.IOQ -95. Read I/O still queued on IO.CLN QIO directive  ( One or more of the packet type code affinities of a task 2 could not be removed because there was still pending Read < I/O for that packet type code. F PIE.SNA -93. Signal not acknowledged Z d The receiver software has failed to acknowledge receipt n of a signal. x IE.NDS -89. Unit is not disconnected from CD driver   When an IO.AWA QIO function was issued the CD driver had  not in fact relinquished the DR11-W vector, so was not in  the 'sleeping' state to be awakened.  IE.NEX -102. Driver non-existant state   This is a driver bug check and should never occur.  On dispatch of the I/O function the driver's own dispatch  table is found to contain an invalid code.  IE.BUG -111. Driver Bug Check   On an interrupt the driver is found to be in an invalid  state. " ,IE.LDN -101. Link Down 6 @ The link protocol between the controlling pieces of J software on either end of the link is not operational. T This may occur because the other machine is down or ^ not running link software. Unless the SF.RSW sub- h function is specified read and write type QIO r functions will terminate with this error condition | when the link is down. A read or write type QIO  function which is being processed may also terminate  with this error condition if a link protocol failure  occurs during the transactions involved in the Signal  or data block transfer.  IE.EME -110. End of message error   The receiver of a block of data detected an error  condition on receiving the end-of-DMA interrupt from  the DR11-W. The error condition indicated by the  receiver in the end of message code can be found  in the second I/O status word.  IE.FHE -59. Fatal Hardware error   On a IO.RLB or IO.WLB QIO function an error has been & detected at the end of the DMA. Either the DR11-W itself 0 has indicated an error condition, or the DMA word count : register was found to have an incorrect final value. D The DR11-W error register contents can be found in the N second word of the I/O status block. X bIE.DAO -13. Data overrun l v On an IO.RLB or IO.WLB function data was transferred from  the sender's buffer to the receiver's buffer but not the  full amount specified on the IO.WLB function. Both  sender and receiver are notified of the truncated  transfer. The actual number of bytes of data transferred  can be found in the second I/O status word.  IE.ABO -15. Request cancelled   If an IO.KIL QIO function is executed, all queued I/O  functions for that CD unit number, will be terminated  with this status.  IE.AST -80. Invalid AST address   The AST address given on a set packet type code affinity  QIO function is not a valid task address.  *IE.REJ -88. Data block rejected by receiver 4 > This status code is returned on an IO.WLB QIO function H when the declared receiver of data blocks of this type R rejects the message. \ fIE.NOD -23. System dynamic memory exhausted p zIE.BAD -1. Bad parameters on QIO  IE.SPC -6. Illegal user buffer   Buffer for data transfer must be in tasks address space,  word aligned.   IE.BYT -19. Odd byte count       .end literal .page