Instead, I'm going to use sockets in a bunch of examples that build.
Ordinary Pipes
Named Pipes/Sockets
Unix Domain Sockets
TCP Sockets/Ports
upstart-socket-bridge
The Simplest
Pipes (|) and redirection (>). We already know how to do that in the shell.
These pipes connect related processes. For example, the children of a single terminal are related.
It's hard to pipe from shell to python (or the reverse) - it's possible, just not easy the first couple times you try.
Shell example:
$ echo Hello, Planet\! | sed 's/Planet/World/' Hello, World!
Here is a good example of pipes in Python 3.
Named Pipes
Slightly more complicated (reference). These use a pipe file that synchronizes the data transfer when both processes are ready. A -> Pipe -> B. This is handy for inter-process communication that does not need to run through dbus, and saves a bit of overhead.
Names pipes are a shared file. They can have many (anonymous) writers, but only a single reader. Shell, Python, and C senders and listeners can interoperate (fairly) smoothly.
Named pipes are blocking. Start the reader first, and it will wait for the writer. Alternately, start the writer first, and it will wait for the reader. Once the writer has finished, the reader will also stop. One read() for one write(). A second read() requires a second write()...and a second write() requires a second read().
Named pipes in shell and C can be reused. Named pipes in Python cannot be reused after close() - they must be unlink() (destroyed) and remade afresh.
Conventionally, the listener is expected to create the pipe. But it's not required.
Processes that use named pipes work in the following order:
- Create the pipe (usually done by listener)
- Open the pipe (both sides)
- Write the pipe
- Writer close the pipe (allows the reader to see the data)
- Listener read the pipe
- Listener close the pipe
- Delete the pipe (usually done by the listener)
Creating and Deleting a Named Pipe:
Shell:
#!/bin/sh mkfifo /tmp/testpipe # Create pipe rm /tmp/testpipe # Delete pipe
Python3:
>>> import os
>>> os.mkfifo("/tmp/testpipe") # Create pipe
>>> os.remove("/tmp/testpipe") # Delete pipe
C:
# include <sys/stat.h>
int main( ){
int result = mkfifo ("/tmp/testpipe", S_IRUSR| S_IWUSR); \\ Create Pipe
unlink ("/tmp/testpipe"); \\ Delete Pipe
return 0;
}
Named Pipe Listeners:
Shell:
#!/bin/sh
pipe=/tmp/testpipe
trap "rm -f $pipe" EXIT
while true
do
read line <$pipe # Read the pipe
echo $line
done
Python3:
>>> import os
>>> pipe = open("/tmp/testpipe", "r")
>>> pipe.read() # You won't read anything until the sender close()
'Hello, World!' # Data from pipe
>>> pipe.close()
C:
#include <fcntl.h>
#include <stdio.h>
int main()
{
char buf[20];
int fd = open("/tmp/testpipe", O_RDONLY); // Open the pipe
read(fd, buf, 20); // Read the message - unblock the writing process
printf("Received: %s\n", buf); // Display the message
close(fd); // Close the pipe
return 0;
}
Named Pipe Senders:
Shell:
#!/bin/sh pipe=/tmp/testpipe [ ! -p $pipe ] && echo "Reader not running" && exit 1 [ "$1" ] && echo "$1" >$pipe || echo "Hello from $$" >$pipe
Python3:
>>> import os
>>> pipe = open("/tmp/testpipe", "w") # Hang here awaiting the receiver to read() once
>>> pipe.write("Hello, World!")
13
>>> pipe.close() # Receiver read() is empty until this happens
C:
#include <fcntl.h>
int main()
{
int fd = open("/tmp/testpipe", O_WRONLY); // Open the pipe
write(fd, "Hi", sizeof("Hi")); // Write "Hi" to pipe
close(fd); // Close the pipe - allow the read
return 0;
}
Unix Domain Sockets
Unix Domain Sockets is a fancy name for a fancy pipe. Like shell pipes, Unix Sockets use a pipe file...Well it *looks* like a pipe file. Actually, data is buffered instead of written to a file. The file is simply an easy way for the processes to find each other.
Unlike shell pipes, Unix sockets are bi-directional, and multiple writers are not anonymous. In Python, sockets are single-use: After a close() they cannot be reopened, and must be deleted and remade. Pipes can only be read() when the sender close(), but sockets can be read() for every send().
Since sockets are bi-directional, it's not a sender/receiver relationship anymore. It's a client/server relationship.
Processes using a Unix Domain Socket usually work in the following order:
- Create a file descriptor (fd) that defines the socket OR look up the fd of an existing socket.
- Server binds to the socket
- Server listens on the socket
- Client connects to the socket
- Server accepts the connection
- Both sides read/write
- Client closes socket (on Python, cannot be reopened - must be reconnected )
- Server closes socket (usually during service shutdown)
- Delete/unlink the file descriptor and socket (if it won't be reused)
Creating and Deleting a Unix Domain Socket:
Shell:
Shell creation can be automatic, for example using netcat:$ nc -lU /tmp/sock # Create socket, among other actions $ rm /tmp/sock # Delete socket
Python3:
>>> import socket, os
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.bind("/tmp/sock") # Create socket
>>> os.remove("/tmp/sock") # Delete socket
C:
#include <sys/socket.h>
#include <sys/un.h>
int main() {
char *path = "/tmp/sock";
struct sockaddr_un addr;
char buf[100];
int cl,rc;
int fd = socket(AF_UNIX, SOCK_STREAM, 0); // Create a file descriptor
addr.sun_family = AF_UNIX; // It's a Unix socket
strcpy(addr.sun_path, path); // Set the path
memset(&addr, 0, sizeof(addr)); // Fill the address array with zeroes (prevent garbage)
bind(fd, (struct sockaddr*)&addr, sizeof(addr)); // Create socket (server)
connect(fd, (struct sockaddr*)&addr, sizeof(addr)); // Connect (client)
unlink(path); // Delete socket
return 0;
}
Unix Domain Socket Servers (Bind to socket):
The server owns (binds) and monitors the socket, and is usually responsible for creating and deleting the socket. (C example)Shell:
$ nc -lU /tmp/sock
Python3:
>>> import socket
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.bind("/tmp/sock") # Listener is first process using socket
>>> tempsocket.listen(1)
>>> conn, addr = tempsocket.accept() # Listener blocks here awaiting sender's connect()
>>> conn.recv(1024).decode('UTF-8') # Listener blocks here awaiting send()
'Hello, World' # Try this a few times to capture each send()
>>> tempsocket.close()
C:
#include <stdio.h>
#include <sys/socket.h>
#include <sys/un.h>
int main() {
char buf[100];
int cl,rc;
int fd = create_socket(socket_path) // Creating the socket is not done here. See above
listen(fd, 5); // listen() is the step after bind()
while (1) {
cl = accept(fd, NULL, NULL);
while ( (rc=read(cl,buf,sizeof(buf))) > 0) {
printf("read %u bytes: %.*s\n", rc, rc, buf);
}
if (rc == 0) {
printf("EOF\n");
close(cl);
}
}
return 0;
}
Unix Domain Socket Clients (Connect to socket):
Clients don't own (bind) the socket. Instead, they connect on-demand.Shell:
$ nc -U /tmp/sock
Python3:
>>> import socket
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.connect("/tmp/sock") # Listener must be active before sender
>>> tempsocket.send(data) # Try this a few times
13
>>> tempsocket.close() # End of stream
C:
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
int main() {
struct sockaddr_un addr;
char buf[100];
int rc;
int fd = create_socket(socket path) // Creating the socket is not done here
// First step after connect()
while( (rc=read(STDIN_FILENO, buf, sizeof(buf))) > 0) write(fd, buf, rc);
return 0;
}
Quick fun with netcat:
The easiest possible bi-directional socket.$ nc -lU /tmp/sock # Enter in Terminal #1 $ nc -U /tmp/sock # Enter in Terminal #2
Type anything on either terminal.
Look! The two communicate!
Here is an example of using netcat to exchange files this way (from the man page)
$ nc -lU /tmp/sock > filename.out # Terminal #1 (sender) $ nc -U /tmp/sock < filename.in # Terminal #2 (receiver)
TCP Sockets (Ports)
TCP ports are very much like Unix sockets. Instead of a file, the data stream is buffered. And Unix sockets are not networkable, but TCP ports are a foundation of the Internet. Ports use client/server, and the server owns (binds) the port that the client connects to. Sockets and Ports use the same shell application (netcat) and Python3 module (socket).
Since there is no file to create or delete, bind() and connect() simply use an IP address and port number.
Processes using a TCP port usually work in the following order (just like a Unix Domain Socket):
- Create a file descriptor (fd) that defines the port
- Server binds to the port
- Server listens on the port
- Client connects to the port
- Server accepts the connection
- Both sides read/write
- Client closes port
- Server closes port (usually during service shutdown)
- Delete/unlink the file descriptor and port (if it won't be reused)
TCP Port Services (Bind to port):
The server owns (binds) and monitors the port.Shell:
$ nc -l 45678 # Port 45678
Python3:
>>> import socket
>>> tempsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> tempsocket.bind(("127.0.0.1", 45678)) # Tuple: (IP Addr, Port)
>>> tempsocket.listen(1)
>>> conn, addr = tempsocket.accept() # Listener blocks here awaiting sender's connect()
>>> conn.recv(1024).decode('UTF-8') # Listener blocks here awaiting send()
'Hello, World' # Try it a few times to catch each send()
>>> tempsocket.close()
C:
#include <stdio.h>
#include <string.h>
#include <netinet/in.h>
int main()
{
int sock_fd, conn_fd; // Socket file descriptor and Connection file descriptor
int port_num = 45678;
socklen_t clilen;
char buffer[256];
struct sockaddr_in serv_addr, cli_addr;
sockfd = socket(AF_INET, SOCK_STREAM, 0); // Define socket type
memset(&serv_addr, 0, sizeof(serv_addr)); // Set array to zeroes
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = INADDR_ANY;
serv_addr.sin_port = htons(port_num);
bind(sock_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)); // Bind
listen(sock_fd,5); // Listen
clilen = sizeof(cli_addr);
conn_fd = accept(sock_fd,
(struct sockaddr *) &cli_addr,
&clilen); // Accept
memset(&buffer, 0, sizeof(buffer)); // Fill buffer with zeroes - prevent garbage
read(conn_fd,buffer,255); // Read
printf("Here is the message: %s\n",buffer);
write(conn_fd,"I got your message",18); // Write
close(conn_fd);
close(sock_fd); // Close
return 0;
}
TCP Port Clients (Connect to port):
The client connects to the port on-demand.Shell:
$ nc localhost 45678 # IP=loopback, Port #45678
Python3:
>>> import socket
>>> tempsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> tempsocket.connect(("127.0.0.1", 45678)) # Tuple: (IP Addr, Port)
>>> data = "Hello, World".encode("utf-8")
>>> tempsocket.send(data) # Try this a few times
12
>>> tempsocket.close()
C:
#include <stdio.h>
#include <string.h>
#include <netdb.h>
int main(int argc, char *argv[])
{
int sock_fd;
int port_num = 45678;
struct sockaddr_in serv_addr;
struct hostent *server;
char buffer[256];
sock_fd = socket(AF_INET, SOCK_STREAM, 0); // Define socket type
server = gethostbyname(port_num); // Lookup port owner
memset(&serv_addr, 0, sizeof(serv_addr)); // Set array to zeroes
serv_addr.sin_family = AF_INET;
bcopy((char *)server->h_addr,
(char *)&serv_addr.sin_addr.s_addr,
server->h_length);
serv_addr.sin_port = htons(port_num);
connect(sock_fd,(struct sockaddr *) &serv_addr,sizeof(serv_addr)); // Connect
printf("Please enter the message: ");
memset(&buffer, 0, sizeof(buffer));
fgets(buffer,255,stdin); // Keyboard input
write(sock_fd,buffer,strlen(buffer)); // Write to server
memset(&buffer, 0, sizeof(buffer)); // Reuse buffer array - clear to prevent garbage
read(sock_fd,buffer,255); // Read response
printf("%s\n",buffer); // Print response
close(sock_fd); // Close
return 0;
}
More fun with netcat
This is the basic way that webservers work.$ echo "Hello, World" | nc -l 45678 # Enter in Terminal #1
$ nc localhost 45678 # Enter in Terminal #2
# OR open http://localhost:45678
# in your favorite web browser
No comments:
Post a Comment