diff --git a/unix/README.md b/unix/README.md new file mode 100644 index 00000000..bc6f6031 --- /dev/null +++ b/unix/README.md @@ -0,0 +1,173 @@ +# Building `sys/unix` + +The sys/unix package provides access to the raw system call interface of the +underlying operating system. See: https://godoc.org/golang.org/x/sys/unix + +Porting Go to a new architecture/OS combination or adding syscalls, types, or +constants to an existing architecture/OS pair requires some manual effort; +however, there are tools that automate much of the process. + +## Build Systems + +There are currently two ways we generate the necessary files. We are currently +migrating the build system to use containers so the builds are reproducible. +This is being done on an OS-by-OS basis. Please update this documentation as +components of the build system change. + +### Old Build System (currently for `GOOS != "Linux" || GOARCH == "sparc64"`) + +The old build system generates the Go files based on the C header files +present on your system. This means that files +for a given GOOS/GOARCH pair must be generated on a system with that OS and +architecture. This also means that the generated code can differ from system +to system, based on differences in the header files. + +To avoid this, if you are using the old build system, only generate the Go +files on an installation with unmodified header files. It is also important to +keep track of which version of the OS the files were generated from (ex. +Darwin 14 vs Darwin 15). This makes it easier to track the progress of changes +and have each OS upgrade correspond to a single change. + +To build the files for your current OS and architecture, make sure GOOS and +GOARCH are set correctly and run `mkall.sh`. This will generate the files for +your specific system. Running `mkall.sh -n` shows the commands that will be run. + +Requirements: bash, perl, go + +### New Build System (currently for `GOOS == "Linux" && GOARCH != "sparc64"`) + +The new build system uses a Docker container to generate the go files directly +from source checkouts of the kernel and various system libraries. This means +that on any platform that supports Docker, all the files using the new build +system can be generated at once, and generated files will not change based on +what the person running the scripts has installed on their computer. + +The OS specific files for the new build system are located in the `${GOOS}` +directory, and the build is coordinated by the `${GOOS}/mkall.go` program. When +the kernel or system library updates, modify the Dockerfile at +`${GOOS}/Dockerfile` to checkout the new release of the source. + +To build all the files under the new build system, you must be on an amd64/Linux +system and have your GOOS and GOARCH set accordingly. Running `mkall.sh` will +then generate all of the files for all of the GOOS/GOARCH pairs in the new build +system. Running `mkall.sh -n` shows the commands that will be run. + +Requirements: bash, perl, go, docker + +## Component files + +This section describes the various files used in the code generation process. +It also contains instructions on how to modify these files to add a new +architecture/OS or to add additional syscalls, types, or constants. Note that +if you are using the new build system, the scripts cannot be called normally. +They must be called from within the docker container. + +### asm files + +The hand-written assembly file at `asm_${GOOS}_${GOARCH}.s` implements system +call dispatch. There are three entry points: +``` + func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) + func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr) + func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) +``` +The first and second are the standard ones; they differ only in how many +arguments can be passed to the kernel. The third is for low-level use by the +ForkExec wrapper. Unlike the first two, it does not call into the scheduler to +let it know that a system call is running. + +When porting Go to an new architecture/OS, this file must be implemented for +each GOOS/GOARCH pair. + +### mksysnum + +Mksysnum is a script located at `${GOOS}/mksysnum.pl` (or `mksysnum_${GOOS}.pl` +for the old system). This script takes in a list of header files containing the +syscall number declarations and parses them to produce the corresponding list of +Go numeric constants. See `zsysnum_${GOOS}_${GOARCH}.go` for the generated +constants. + +Adding new syscall numbers is mostly done by running the build on a sufficiently +new installation of the target OS (or updating the source checkouts for the +new build system). However, depending on the OS, you make need to update the +parsing in mksysnum. + +### mksyscall.pl + +The `syscall.go`, `syscall_${GOOS}.go`, `syscall_${GOOS}_${GOARCH}.go` are +hand-written Go files which implement system calls (for unix, the specific OS, +or the specific OS/Architecture pair respectively) that need special handling +and list `//sys` comments giving prototypes for ones that can be generated. + +The mksyscall.pl script takes the `//sys` and `//sysnb` comments and converts +them into syscalls. This requires the name of the prototype in the comment to +match a syscall number in the `zsysnum_${GOOS}_${GOARCH}.go` file. The function +prototype can be exported (capitalized) or not. + +Adding a new syscall often just requires adding a new `//sys` function prototype +with the desired arguments and a capitalized name so it is exported. However, if +you want the interface to the syscall to be different, often one will make an +unexported `//sys` prototype, an then write a custom wrapper in +`syscall_${GOOS}.go`. + +### types files + +For each OS, there is a hand-written Go file at `${GOOS}/types.go` (or +`types_${GOOS}.go` on the old system). This file includes standard C headers and +creates Go type aliases to the corresponding C types. The file is then fed +through godef to get the Go compatible definitions. Finally, the generated code +is fed though mkpost.go to format the code correctly and remove any hidden or +private identifiers. This cleaned-up code is written to +`ztypes_${GOOS}_${GOARCH}.go`. + +The hardest part about preparing this file is figuring out which headers to +include and which symbols need to be `#define`d to get the actual data +structures that pass through to the kernel system calls. Some C libraries +preset alternate versions for binary compatibility and translate them on the +way in and out of system calls, but there is almost always a `#define` that can +get the real ones. +See `types_darwin.go` and `linux/types.go` for examples. + +To add a new type, add in the necessary include statement at the top of the +file (if it is not already there) and add in a type alias line. Note that if +your type is significantly different on different architectures, you may need +some `#if/#elif` macros in your include statements. + +### mkerrors.sh + +This script is used to generate the system's various constants. This doesn't +just include the error numbers and error strings, but also the signal numbers +an a wide variety of miscellaneous constants. The constants come from the list +of include files in the `includes_${uname}` variable. A regex then picks out +the desired `#define` statements, and generates the corresponding Go constants. +The error numbers and strings are generated from `#include `, and the +signal numbers and strings are generated from `#include `. All of +these constants are written to `zerrors_${GOOS}_${GOARCH}.go` via a C program, +`_errors.c`, which prints out all the constants. + +To add a constant, add the header that includes it to the appropriate variable. +Then, edit the regex (if necessary) to match the desired constant. Avoid making +the regex too broad to avoid matching unintended constants. + + +## Generated files + +### `zerror_${GOOS}_${GOARCH}.go` + +A file containing all of the system's generated error numbers, error strings, +signal numbers, and constants. Generated by `mkerrors.sh` (see above). + +### `zsyscall_${GOOS}_${GOARCH}.go` + +A file containing all the generated syscalls for a specific GOOS and GOARCH. +Generated by `mksyscall.pl` (see above). + +### `zsysnum_${GOOS}_${GOARCH}.go` + +A list of numeric constants for all the syscall number of the specific GOOS +and GOARCH. Generated by mksysnum (see above). + +### `ztypes_${GOOS}_${GOARCH}.go` + +A file containing Go types for passing into (or returning from) syscalls. +Generated by godefs and the types file (see above). diff --git a/unix/linux/Dockerfile b/unix/linux/Dockerfile new file mode 100644 index 00000000..4397143c --- /dev/null +++ b/unix/linux/Dockerfile @@ -0,0 +1,48 @@ +FROM ubuntu:16.04 + +# Dependencies to get the git sources and go binaries +RUN apt-get update && apt-get install -y \ + curl \ + git \ + && rm -rf /var/lib/apt/lists/* + +# Get the git sources. If not cached, this takes O(5 minutes). +WORKDIR /git +RUN git config --global advice.detachedHead false +# Linux Kernel: Released 19 Feb 2017 +RUN git clone --branch v4.10 --depth 1 https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux +# GNU C library: Released 05 Feb 2017 (we should try to get a secure way to clone this) +RUN git clone --branch glibc-2.25 --depth 1 git://sourceware.org/git/glibc.git + +# Get Go 1.8 (https://github.com/docker-library/golang/blob/master/1.8/Dockerfile) +ENV GOLANG_VERSION 1.8 +ENV GOLANG_DOWNLOAD_URL https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz +ENV GOLANG_DOWNLOAD_SHA256 53ab94104ee3923e228a2cb2116e5e462ad3ebaeea06ff04463479d7f12d27ca + +RUN curl -fsSL "$GOLANG_DOWNLOAD_URL" -o golang.tar.gz \ + && echo "$GOLANG_DOWNLOAD_SHA256 golang.tar.gz" | sha256sum -c - \ + && tar -C /usr/local -xzf golang.tar.gz \ + && rm golang.tar.gz + +ENV PATH /usr/local/go/bin:$PATH + +# Linux and Glibc build dependencies +RUN apt-get update && apt-get install -y \ + gawk make python \ + gcc gcc-multilib \ + gettext texinfo \ + && rm -rf /var/lib/apt/lists/* +# Emulator and cross compilers +RUN apt-get update && apt-get install -y \ + qemu \ + gcc-aarch64-linux-gnu gcc-arm-linux-gnueabi \ + gcc-mips-linux-gnu gcc-mips64-linux-gnuabi64 \ + gcc-mips64el-linux-gnuabi64 gcc-mipsel-linux-gnu \ + gcc-powerpc64-linux-gnu gcc-powerpc64le-linux-gnu \ + gcc-s390x-linux-gnu gcc-sparc64-linux-gnu \ + && rm -rf /var/lib/apt/lists/* + +# Let the scripts know they are in the docker environment +ENV GOLANG_SYS_BUILD docker +WORKDIR /build +ENTRYPOINT ["go", "run", "linux/mkall.go", "/git/linux", "/git/glibc"] diff --git a/unix/linux/mkall.go b/unix/linux/mkall.go new file mode 100644 index 00000000..429754f1 --- /dev/null +++ b/unix/linux/mkall.go @@ -0,0 +1,379 @@ +// Copyright 2017 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// linux/mkall.go - Generates all Linux zsysnum, zsyscall, zerror, and ztype +// files for all 11 linux architectures supported by the go compiler. See +// README.md for more information about the build system. + +// To run it you must have a git checkout of the Linux kernel and glibc. Once +// the appropriate sources are ready, the program is run as: +// go run linux/mkall.go + +// +build ignore + +package main + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" +) + +// These will be paths to the appropriate source directories. +var LinuxDir string +var GlibcDir string + +const TempDir = "/tmp" +const IncludeDir = TempDir + "/include" // To hold our C headers +const BuildDir = TempDir + "/build" // To hold intermediate build files + +const GOOS = "linux" // Only for Linux targets +const BuildArch = "amd64" // Must be built on this architecture +const MinKernel = "2.6.23" // https://golang.org/doc/install#requirements + +type target struct { + GoArch string // Architecture name according to Go + LinuxArch string // Architecture name according to the Linux Kernel + GNUArch string // Architecture name according to GNU tools (https://wiki.debian.org/Multiarch/Tuples) + BigEndian bool // Default Little Endian + SignedChar bool // Is -fsigned-char needed (default no) + Bits int +} + +// List of the 11 Linux targets supported by the go compiler. sparc64 is not +// currently supported, though a port is in progress. +var targets = []target{ + { + GoArch: "386", + LinuxArch: "x86", + GNUArch: "i686-linux-gnu", // Note "i686" not "i386" + Bits: 32, + }, + { + GoArch: "amd64", + LinuxArch: "x86", + GNUArch: "x86_64-linux-gnu", + Bits: 64, + }, + { + GoArch: "arm64", + LinuxArch: "arm64", + GNUArch: "aarch64-linux-gnu", + SignedChar: true, + Bits: 64, + }, + { + GoArch: "arm", + LinuxArch: "arm", + GNUArch: "arm-linux-gnueabi", + Bits: 32, + }, + { + GoArch: "mips", + LinuxArch: "mips", + GNUArch: "mips-linux-gnu", + BigEndian: true, + Bits: 32, + }, + { + GoArch: "mipsle", + LinuxArch: "mips", + GNUArch: "mipsel-linux-gnu", + Bits: 32, + }, + { + GoArch: "mips64", + LinuxArch: "mips", + GNUArch: "mips64-linux-gnuabi64", + BigEndian: true, + Bits: 64, + }, + { + GoArch: "mips64le", + LinuxArch: "mips", + GNUArch: "mips64el-linux-gnuabi64", + Bits: 64, + }, + { + GoArch: "ppc64", + LinuxArch: "powerpc", + GNUArch: "powerpc64-linux-gnu", + BigEndian: true, + Bits: 64, + }, + { + GoArch: "ppc64le", + LinuxArch: "powerpc", + GNUArch: "powerpc64le-linux-gnu", + Bits: 64, + }, + { + GoArch: "s390x", + LinuxArch: "s390", + GNUArch: "s390x-linux-gnu", + BigEndian: true, + SignedChar: true, + Bits: 64, + }, + // { + // GoArch: "sparc64", + // LinuxArch: "sparc", + // GNUArch: "sparc64-linux-gnu", + // BigEndian: true, + // Bits: 64, + // }, +} + +func main() { + if runtime.GOOS != GOOS || runtime.GOARCH != BuildArch { + fmt.Printf("Build system has GOOS_GOARCH = %s_%s, need %s_%s\n", + runtime.GOOS, runtime.GOARCH, GOOS, BuildArch) + return + } + + // Check that we are using the new build system if we should + if os.Getenv("GOLANG_SYS_BUILD") != "docker" { + fmt.Println("In the new build system, mkall.go should not be called directly.") + fmt.Println("See README.md") + return + } + + // Parse the command line options + if len(os.Args) != 3 { + fmt.Println("USAGE: go run linux/mkall.go ") + return + } + LinuxDir = os.Args[1] + GlibcDir = os.Args[2] + + for _, t := range targets { + fmt.Printf("----- GENERATING: %s -----\n", t.GoArch) + if err := t.generateFiles(); err != nil { + fmt.Printf("%v\n***** FAILURE: %s *****\n\n", err, t.GoArch) + } else { + fmt.Printf("----- SUCCESS: %s -----\n\n", t.GoArch) + } + } +} + +// Makes an exec.Cmd with Stderr attached to os.Stderr +func makeCommand(name string, args ...string) *exec.Cmd { + cmd := exec.Command(name, args...) + cmd.Stderr = os.Stderr + return cmd +} + +// Runs the command, pipes output to a formatter, pipes that to an output file. +func (t *target) commandFormatOutput(formatter string, outputFile string, + name string, args ...string) (err error) { + mainCmd := makeCommand(name, args...) + + fmtCmd := makeCommand(formatter) + if formatter == "mkpost" { + fmtCmd = makeCommand("go", "run", "mkpost.go") + // Set GOARCH_TARGET so mkpost knows what GOARCH is.. + fmtCmd.Env = append(os.Environ(), "GOARCH_TARGET="+t.GoArch) + // Set GOARCH to host arch for mkpost, so it can run natively. + for i, s := range fmtCmd.Env { + if strings.HasPrefix(s, "GOARCH=") { + fmtCmd.Env[i] = "GOARCH=" + BuildArch + } + } + } + + // mainCmd | fmtCmd > outputFile + if fmtCmd.Stdin, err = mainCmd.StdoutPipe(); err != nil { + return + } + if fmtCmd.Stdout, err = os.Create(outputFile); err != nil { + return + } + + // Make sure the formatter eventually closes + if err = fmtCmd.Start(); err != nil { + return + } + defer func() { + fmtErr := fmtCmd.Wait() + if err == nil { + err = fmtErr + } + }() + + return mainCmd.Run() +} + +// Generates all the files for a Linux target +func (t *target) generateFiles() error { + // Setup environment variables + os.Setenv("GOOS", GOOS) + os.Setenv("GOARCH", t.GoArch) + + // Get appropriate compiler and emulator (unless on x86) + if t.LinuxArch != "x86" { + // Check/Setup cross compiler + compiler := t.GNUArch + "-gcc" + if _, err := exec.LookPath(compiler); err != nil { + return err + } + os.Setenv("CC", compiler) + + // Check/Setup emulator (usually first component of GNUArch) + qemuArchName := t.GNUArch[:strings.Index(t.GNUArch, "-")] + if t.LinuxArch == "powerpc" { + qemuArchName = t.GoArch + } + os.Setenv("GORUN", "qemu-"+qemuArchName) + } else { + os.Setenv("CC", "gcc") + } + + // Make the include directory and fill it with headers + if err := os.MkdirAll(IncludeDir, os.ModePerm); err != nil { + return err + } + defer os.RemoveAll(IncludeDir) + if err := t.makeHeaders(); err != nil { + return fmt.Errorf("could not make header files: %v", err) + } + fmt.Println("header files generated") + + // Make each of the four files + if err := t.makeZSysnumFile(); err != nil { + return fmt.Errorf("could not make zsysnum file: %v", err) + } + fmt.Println("zsysnum file generated") + + if err := t.makeZSyscallFile(); err != nil { + return fmt.Errorf("could not make zsyscall file: %v", err) + } + fmt.Println("zsyscall file generated") + + if err := t.makeZTypesFile(); err != nil { + return fmt.Errorf("could not make ztypes file: %v", err) + } + fmt.Println("ztypes file generated") + + if err := t.makeZErrorsFile(); err != nil { + return fmt.Errorf("could not make zerrors file: %v", err) + } + fmt.Println("zerrors file generated") + + return nil +} + +// Create the Linux and glibc headers in the include directory. +func (t *target) makeHeaders() error { + // Make the Linux headers we need for this architecture + linuxMake := makeCommand("make", "headers_install", "ARCH="+t.LinuxArch, "INSTALL_HDR_PATH="+TempDir) + linuxMake.Dir = LinuxDir + if err := linuxMake.Run(); err != nil { + return err + } + + // A Temporary build directory for glibc + if err := os.MkdirAll(BuildDir, os.ModePerm); err != nil { + return err + } + defer os.RemoveAll(BuildDir) + + // Make the glibc headers we need for this architecture + confScript := filepath.Join(GlibcDir, "configure") + glibcConf := makeCommand(confScript, "--prefix="+TempDir, "--host="+t.GNUArch, "--enable-kernel="+MinKernel) + glibcConf.Dir = BuildDir + if err := glibcConf.Run(); err != nil { + return err + } + glibcMake := makeCommand("make", "install-headers") + glibcMake.Dir = BuildDir + if err := glibcMake.Run(); err != nil { + return err + } + // We only need an empty stubs file + stubsFile := filepath.Join(IncludeDir, "gnu/stubs.h") + if file, err := os.Create(stubsFile); err != nil { + return err + } else { + file.Close() + } + + return nil +} + +// makes the zsysnum_linux_$GOARCH.go file +func (t *target) makeZSysnumFile() error { + zsysnumFile := fmt.Sprintf("zsysnum_linux_%s.go", t.GoArch) + unistdFile := filepath.Join(IncludeDir, "asm/unistd.h") + + args := append(t.cFlags(), unistdFile) + return t.commandFormatOutput("gofmt", zsysnumFile, "linux/mksysnum.pl", args...) +} + +// makes the zsyscall_linux_$GOARCH.go file +func (t *target) makeZSyscallFile() error { + zsyscallFile := fmt.Sprintf("zsyscall_linux_%s.go", t.GoArch) + // Find the correct architecture syscall file (might end with x.go) + archSyscallFile := fmt.Sprintf("syscall_linux_%s.go", t.GoArch) + if _, err := os.Stat(archSyscallFile); os.IsNotExist(err) { + shortArch := strings.TrimSuffix(t.GoArch, "le") + archSyscallFile = fmt.Sprintf("syscall_linux_%sx.go", shortArch) + } + + args := append(t.mksyscallFlags(), "-tags", "linux,"+t.GoArch, + "syscall_linux.go", archSyscallFile) + return t.commandFormatOutput("gofmt", zsyscallFile, "./mksyscall.pl", args...) +} + +// makes the zerrors_linux_$GOARCH.go file +func (t *target) makeZErrorsFile() error { + zerrorsFile := fmt.Sprintf("zerrors_linux_%s.go", t.GoArch) + + return t.commandFormatOutput("gofmt", zerrorsFile, "./mkerrors.sh", t.cFlags()...) +} + +// makes the ztypes_linux_$GOARCH.go file +func (t *target) makeZTypesFile() error { + ztypesFile := fmt.Sprintf("ztypes_linux_%s.go", t.GoArch) + + args := []string{"tool", "cgo", "-godefs", "--"} + args = append(args, t.cFlags()...) + args = append(args, "linux/types.go") + return t.commandFormatOutput("mkpost", ztypesFile, "go", args...) +} + +// Flags that should be given to gcc and cgo for this target +func (t *target) cFlags() []string { + // Compile statically to avoid cross-architecture dynamic linking. + flags := []string{"-Wall", "-Werror", "-static", "-I" + IncludeDir} + + // Architecture-specific flags + if t.SignedChar { + flags = append(flags, "-fsigned-char") + } + if t.LinuxArch == "x86" { + flags = append(flags, fmt.Sprintf("-m%d", t.Bits)) + } + + return flags +} + +// Flags that should be given to mksyscall for this target +func (t *target) mksyscallFlags() (flags []string) { + if t.Bits == 32 { + if t.BigEndian { + flags = append(flags, "-b32") + } else { + flags = append(flags, "-l32") + } + } + + // This flag menas a 64-bit value should use (even, odd)-pair. + if t.GoArch == "arm" || (t.LinuxArch == "mips" && t.Bits == 32) { + flags = append(flags, "-arm") + } + return +} diff --git a/unix/mksysnum_linux.pl b/unix/linux/mksysnum.pl similarity index 78% rename from unix/mksysnum_linux.pl rename to unix/linux/mksysnum.pl index 872ae8c5..63fd800b 100755 --- a/unix/mksysnum_linux.pl +++ b/unix/linux/mksysnum.pl @@ -10,11 +10,18 @@ if($ENV{'GOARCH'} eq "" || $ENV{'GOOS'} eq "") { exit 1; } -my $command = "mksysnum_linux.pl ". join(' ', @ARGV); +# Check that we are using the new build system if we should +if($ENV{'GOLANG_SYS_BUILD'} ne "docker") { + print STDERR "In the new build system, mksysnum should not be called directly.\n"; + print STDERR "See README.md\n"; + exit 1; +} + +my $command = "$0 ". join(' ', @ARGV); print <){ +open(CC, "$ENV{'CC'} -E -dD @ARGV |") || die "can't run $ENV{'CC'}"; +while(){ if(/^#define __NR_Linux\s+([0-9]+)/){ # mips/mips64: extract offset $offset = $1; diff --git a/unix/types_linux.go b/unix/linux/types.go similarity index 88% rename from unix/types_linux.go rename to unix/linux/types.go index ae79779e..2f8b7cc0 100644 --- a/unix/types_linux.go +++ b/unix/linux/types.go @@ -5,7 +5,7 @@ // +build ignore /* -Input to cgo -godefs. See also mkerrors.sh and mkall.sh +Input to cgo -godefs. See README.md */ // +godefs map struct_in_addr [4]byte /* in_addr */ @@ -20,7 +20,6 @@ package unix #define _GNU_SOURCE #include -#include #include #include #include @@ -36,13 +35,11 @@ package unix #include #include #include -#include #include #include #include #include #include -#include #include #include #include @@ -52,16 +49,64 @@ package unix #include #include #include +#include #include #include #include #include -#include -#include #include #include #include +// On mips64, the glibc stat and kernel stat do not agree +#if (defined(__mips__) && _MIPS_SIM == _MIPS_SIM_ABI64) + +// Use the stat defined by the kernel with a few modifications. These are: +// * The time fields (like st_atime and st_atimensec) use the timespec +// struct (like st_atim) for consitancy with the glibc fields. +// * The padding fields get different names to not break compatibility. +// * st_blocks is signed, again for compatibility. +struct stat { + unsigned int st_dev; + unsigned int st_pad1[3]; // Reserved for st_dev expansion + + unsigned long st_ino; + + mode_t st_mode; + __u32 st_nlink; + + uid_t st_uid; + gid_t st_gid; + + unsigned int st_rdev; + unsigned int st_pad2[3]; // Reserved for st_rdev expansion + + off_t st_size; + + // These are declared as speperate fields in the kernel. Here we use + // the timespec struct for consistancy with the other stat structs. + struct timespec st_atim; + struct timespec st_mtim; + struct timespec st_ctim; + + unsigned int st_blksize; + unsigned int st_pad4; + + long st_blocks; +}; + +// These are needed because we do not include fcntl.h or sys/types.h +#include +#include + +#else + +// Use the stat defined by glibc +#include +#include + +#endif + #ifdef TCSETS2 // On systems that have "struct termios2" use this as type Termios. typedef struct termios2 termios_t; @@ -87,6 +132,13 @@ struct sockaddr_any { char pad[sizeof(union sockaddr_all) - sizeof(struct sockaddr)]; }; +// copied from /usr/include/bluetooth/hci.h +struct sockaddr_hci { + sa_family_t hci_family; + unsigned short hci_dev; + unsigned short hci_channel; +};; + // copied from /usr/include/linux/un.h struct my_sockaddr_un { sa_family_t sun_family; @@ -421,11 +473,11 @@ const SizeofInotifyEvent = C.sizeof_struct_inotify_event type PtraceRegs C.PtraceRegs // Structures contained in PtraceRegs on s390x (exported by mkpost.go) -type ptracePsw C.ptracePsw +type PtracePsw C.ptracePsw -type ptraceFpregs C.ptraceFpregs +type PtraceFpregs C.ptraceFpregs -type ptracePer C.ptracePer +type PtracePer C.ptracePer // Misc diff --git a/unix/mkall.sh b/unix/mkall.sh index c1fc2adb..a1a27ae7 100755 --- a/unix/mkall.sh +++ b/unix/mkall.sh @@ -3,75 +3,9 @@ # Use of this source code is governed by a BSD-style # license that can be found in the LICENSE file. -# The unix package provides access to the raw system call -# interface of the underlying operating system. Porting Go to -# a new architecture/operating system combination requires -# some manual effort, though there are tools that automate -# much of the process. The auto-generated files have names -# beginning with z. -# -# This script runs or (given -n) prints suggested commands to generate z files -# for the current system. Running those commands is not automatic. -# This script is documentation more than anything else. -# -# * asm_${GOOS}_${GOARCH}.s -# -# This hand-written assembly file implements system call dispatch. -# There are three entry points: -# -# func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr); -# func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr); -# func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr); -# -# The first and second are the standard ones; they differ only in -# how many arguments can be passed to the kernel. -# The third is for low-level use by the ForkExec wrapper; -# unlike the first two, it does not call into the scheduler to -# let it know that a system call is running. -# -# * syscall_${GOOS}.go -# -# This hand-written Go file implements system calls that need -# special handling and lists "//sys" comments giving prototypes -# for ones that can be auto-generated. Mksyscall reads those -# comments to generate the stubs. -# -# * syscall_${GOOS}_${GOARCH}.go -# -# Same as syscall_${GOOS}.go except that it contains code specific -# to ${GOOS} on one particular architecture. -# -# * types_${GOOS}.c -# -# This hand-written C file includes standard C headers and then -# creates typedef or enum names beginning with a dollar sign -# (use of $ in variable names is a gcc extension). The hardest -# part about preparing this file is figuring out which headers to -# include and which symbols need to be #defined to get the -# actual data structures that pass through to the kernel system calls. -# Some C libraries present alternate versions for binary compatibility -# and translate them on the way in and out of system calls, but -# there is almost always a #define that can get the real ones. -# See types_darwin.c and types_linux.c for examples. -# -# * zerror_${GOOS}_${GOARCH}.go -# -# This machine-generated file defines the system's error numbers, -# error strings, and signal numbers. The generator is "mkerrors.sh". -# Usually no arguments are needed, but mkerrors.sh will pass its -# arguments on to godefs. -# -# * zsyscall_${GOOS}_${GOARCH}.go -# -# Generated by mksyscall.pl; see syscall_${GOOS}.go above. -# -# * zsysnum_${GOOS}_${GOARCH}.go -# -# Generated by mksysnum_${GOOS}. -# -# * ztypes_${GOOS}_${GOARCH}.go -# -# Generated by godefs; see types_${GOOS}.c above. +# This script runs or (given -n) prints suggested commands to generate files for +# the Architecture/OS specified by the GOARCH and GOOS environment variables. +# See README.md for more information about how the build system works. GOOSARCH="${GOOS}_${GOARCH}" @@ -84,6 +18,7 @@ zsysctl="zsysctl_$GOOSARCH.go" mksysnum= mktypes= run="sh" +cmd="" case "$1" in -syscalls) @@ -98,6 +33,7 @@ case "$1" in ;; -n) run="cat" + cmd="echo" shift esac @@ -109,6 +45,14 @@ case "$#" in exit 2 esac +if [[ "$GOOS" -eq "linux" ]] && [[ "$GOARCH" != "sparc64" ]]; then + # Use then new build system + # Files generated through docker (use $cmd so you can Ctl-C the build or run) + $cmd docker build --tag generate:$GOOS $GOOS + $cmd docker run --interactive --tty --volume $(dirname "$(readlink -f "$0")"):/build generate:$GOOS + exit +fi + GOOSARCH_in=syscall_$GOOSARCH.go case "$GOOSARCH" in _* | *_ | _) @@ -167,64 +111,6 @@ freebsd_arm) # API consistent across over platforms. mktypes="GOARCH=$GOARCH go tool cgo -godefs -- -fsigned-char" ;; -linux_386) - mkerrors="$mkerrors -m32" - mksyscall="./mksyscall.pl -l32" - mksysnum="./mksysnum_linux.pl /usr/include/asm/unistd_32.h" - mktypes="GOARCH=$GOARCH go tool cgo -godefs" - ;; -linux_amd64) - unistd_h=$(ls -1 /usr/include/asm/unistd_64.h /usr/include/x86_64-linux-gnu/asm/unistd_64.h 2>/dev/null | head -1) - if [ "$unistd_h" = "" ]; then - echo >&2 cannot find unistd_64.h - exit 1 - fi - mkerrors="$mkerrors -m64" - mksysnum="./mksysnum_linux.pl $unistd_h" - mktypes="GOARCH=$GOARCH go tool cgo -godefs" - ;; -linux_arm) - mkerrors="$mkerrors" - mksyscall="./mksyscall.pl -l32 -arm" - mksysnum="curl -s 'http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/arch/arm/include/uapi/asm/unistd.h' | ./mksysnum_linux.pl -" - mktypes="GOARCH=$GOARCH go tool cgo -godefs" - ;; -linux_arm64) - unistd_h=$(ls -1 /usr/include/asm/unistd.h /usr/include/asm-generic/unistd.h 2>/dev/null | head -1) - if [ "$unistd_h" = "" ]; then - echo >&2 cannot find unistd_64.h - exit 1 - fi - mksysnum="./mksysnum_linux.pl $unistd_h" - # Let the type of C char be signed for making the bare syscall - # API consistent across over platforms. - mktypes="GOARCH=$GOARCH go tool cgo -godefs -- -fsigned-char" - ;; -linux_ppc64) - GOOSARCH_in=syscall_linux_ppc64x.go - unistd_h=/usr/include/asm/unistd.h - mkerrors="$mkerrors -m64" - mksysnum="./mksysnum_linux.pl $unistd_h" - mktypes="GOARCH=$GOARCH go tool cgo -godefs" - ;; -linux_ppc64le) - GOOSARCH_in=syscall_linux_ppc64x.go - unistd_h=/usr/include/powerpc64le-linux-gnu/asm/unistd.h - mkerrors="$mkerrors -m64" - mksysnum="./mksysnum_linux.pl $unistd_h" - mktypes="GOARCH=$GOARCH go tool cgo -godefs" - ;; -linux_s390x) - GOOSARCH_in=syscall_linux_s390x.go - unistd_h=/usr/include/asm/unistd.h - mkerrors="$mkerrors -m64" - mksysnum="./mksysnum_linux.pl $unistd_h" - # Let the type of C char be signed to make the bare sys - # API more consistent between platforms. - # This is a deliberate departure from the way the syscall - # package generates its version of the types file. - mktypes="GOARCH=$GOARCH go tool cgo -godefs -- -fsigned-char" - ;; linux_sparc64) GOOSARCH_in=syscall_linux_sparc64.go unistd_h=/usr/include/sparc64-linux-gnu/asm/unistd.h @@ -288,7 +174,6 @@ esac if [ -n "$mksysctl" ]; then echo "$mksysctl |gofmt >$zsysctl"; fi if [ -n "$mksysnum" ]; then echo "$mksysnum |gofmt >zsysnum_$GOOSARCH.go"; fi if [ -n "$mktypes" ]; then - echo "echo // +build $GOARCH,$GOOS > ztypes_$GOOSARCH.go"; - echo "$mktypes types_$GOOS.go | go run mkpost.go >>ztypes_$GOOSARCH.go"; + echo "$mktypes types_$GOOS.go | go run mkpost.go > ztypes_$GOOSARCH.go"; fi ) | $run diff --git a/unix/mkerrors.sh b/unix/mkerrors.sh index 8da63576..23648b5b 100755 --- a/unix/mkerrors.sh +++ b/unix/mkerrors.sh @@ -16,6 +16,15 @@ if test -z "$GOARCH" -o -z "$GOOS"; then exit 1 fi +# Check that we are using the new build system if we should +if [[ "$GOOS" -eq "linux" ]] && [[ "$GOARCH" != "sparc64" ]]; then + if [[ "$GOLANG_SYS_BUILD" -ne "docker" ]]; then + echo 1>&2 "In the new build system, mkerrors should not be called directly." + echo 1>&2 "See README.md" + exit 1 + fi +fi + CC=${CC:-cc} if [[ "$GOOS" -eq "solaris" ]]; then @@ -102,6 +111,36 @@ includes_Linux=' #endif #define _GNU_SOURCE +// is broken on powerpc64, as it fails to include definitions of +// these structures. We just include them copied from . +#if defined(__powerpc__) +struct sgttyb { + char sg_ispeed; + char sg_ospeed; + char sg_erase; + char sg_kill; + short sg_flags; +}; + +struct tchars { + char t_intrc; + char t_quitc; + char t_startc; + char t_stopc; + char t_eofc; + char t_brkc; +}; + +struct ltchars { + char t_suspc; + char t_dsuspc; + char t_rprntc; + char t_flushc; + char t_werasc; + char t_lnextc; +}; +#endif + #include #include #include @@ -391,7 +430,7 @@ echo '#include ' | $CC -x c - -E -dM $ccflags | sort >_signal.grep echo '// mkerrors.sh' "$@" -echo '// MACHINE GENERATED BY THE COMMAND ABOVE; DO NOT EDIT' +echo '// Code generated by the command above; see README.md. DO NOT EDIT.' echo echo "// +build ${GOARCH},${GOOS}" echo @@ -453,7 +492,7 @@ intcmp(const void *a, const void *b) int main(void) { - int i, j, e; + int i, e; char buf[1024], *p; printf("\n\n// Error table\n"); diff --git a/unix/mkpost.go b/unix/mkpost.go index ed50d902..d3ff659b 100644 --- a/unix/mkpost.go +++ b/unix/mkpost.go @@ -8,10 +8,11 @@ // modify the generated types. It is used to clean up // the sys API in an architecture specific manner. // -// mkpost is run after cgo -godefs by mkall.sh. +// mkpost is run after cgo -godefs; see README.md. package main import ( + "bytes" "fmt" "go/format" "io/ioutil" @@ -21,42 +22,67 @@ import ( ) func main() { + // Get the OS and architecture (using GOARCH_TARGET if it exists) + goos := os.Getenv("GOOS") + goarch := os.Getenv("GOARCH_TARGET") + if goarch == "" { + goarch = os.Getenv("GOARCH") + } + // Check that we are using the new build system if we should be. + if goos == "linux" && goarch != "sparc64" { + if os.Getenv("GOLANG_SYS_BUILD") != "docker" { + os.Stderr.WriteString("In the new build system, mkpost should not be called directly.\n") + os.Stderr.WriteString("See README.md\n") + os.Exit(1) + } + } + b, err := ioutil.ReadAll(os.Stdin) if err != nil { log.Fatal(err) } - s := string(b) - goarch := os.Getenv("GOARCH") - goos := os.Getenv("GOOS") + // If we have empty Ptrace structs, we should delete them. Only s390x emits + // nonempty Ptrace structs. + ptraceRexexp := regexp.MustCompile(`type Ptrace((Psw|Fpregs|Per) struct {\s*})`) + b = ptraceRexexp.ReplaceAll(b, nil) + + // Replace the control_regs union with a blank identifier for now. + controlRegsRegex := regexp.MustCompile(`(Control_regs)\s+\[0\]uint64`) + b = controlRegsRegex.ReplaceAll(b, []byte("_ [0]uint64")) + + // Remove fields that are added by glibc + // Note that this is unstable as the identifers are private. + removeFieldsRegex := regexp.MustCompile(`X__glibc\S*`) + b = removeFieldsRegex.ReplaceAll(b, []byte("_")) + + // We refuse to export private fields on s390x if goarch == "s390x" && goos == "linux" { - // Export the types of PtraceRegs fields. - re := regexp.MustCompile("ptrace(Psw|Fpregs|Per)") - s = re.ReplaceAllString(s, "Ptrace$1") + // Remove cgo padding fields + removeFieldsRegex := regexp.MustCompile(`Pad_cgo_\d+`) + b = removeFieldsRegex.ReplaceAll(b, []byte("_")) - // Replace padding fields inserted by cgo with blank identifiers. - re = regexp.MustCompile("Pad_cgo[A-Za-z0-9_]*") - s = re.ReplaceAllString(s, "_") - - // Replace other unwanted fields with blank identifiers. - re = regexp.MustCompile("X_[A-Za-z0-9_]*") - s = re.ReplaceAllString(s, "_") - - // Replace the control_regs union with a blank identifier for now. - re = regexp.MustCompile("(Control_regs)\\s+\\[0\\]uint64") - s = re.ReplaceAllString(s, "_ [0]uint64") + // Remove padding, hidden, or unused fields + removeFieldsRegex = regexp.MustCompile(`X_\S+`) + b = removeFieldsRegex.ReplaceAll(b, []byte("_")) } + // Remove the first line of warning from cgo + b = b[bytes.IndexByte(b, '\n')+1:] + // Modify the command in the header to include: + // mkpost, our own warning, and a build tag. + replacement := fmt.Sprintf(`$1 | go run mkpost.go +// Code generated by the command above; see README.md. DO NOT EDIT. + +// +build %s,%s`, goarch, goos) + cgoCommandRegex := regexp.MustCompile(`(cgo -godefs .*)`) + b = cgoCommandRegex.ReplaceAll(b, []byte(replacement)) + // gofmt - b, err = format.Source([]byte(s)) + b, err = format.Source(b) if err != nil { log.Fatal(err) } - // Append this command to the header to show where the new file - // came from. - re := regexp.MustCompile("(cgo -godefs [a-zA-Z0-9_]+\\.go.*)") - b = re.ReplaceAll(b, []byte("$1 | go run mkpost.go")) - - fmt.Printf("%s", b) + os.Stdout.Write(b) } diff --git a/unix/mksyscall.pl b/unix/mksyscall.pl index 34f8ef82..eb4b56ff 100755 --- a/unix/mksyscall.pl +++ b/unix/mksyscall.pl @@ -69,6 +69,16 @@ if($ARGV[0] =~ /^-/) { exit 1; } +# Check that we are using the new build system if we should +if($ENV{'GOOS'} eq "linux" || $ENV{'GOARCH'} ne "sparc64") { + if($ENV{'GOLANG_SYS_BUILD'} ne "docker") { + print STDERR "In the new build system, mksyscall should not be called directly.\n"; + print STDERR "See README.md\n"; + exit 1; + } +} + + sub parseparamlist($) { my ($list) = @_; $list =~ s/^\s*//; @@ -300,7 +310,7 @@ if($errors) { print <